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FOREWORD 


The  U.S.  Army  is  facing  the  challenge  of  meeting  complex  training  requirements  with 
increasingly  constrained  resources.  As  a  result.  Army  trainers  are  exploring  emerging 
technologies  and  innovative  techniques  that  can  be  used  to  achieve  training  goals.  Part  of  this 
exploration  has  been  accomplished  through  the  Force  XXI  Training  Program  (FXXITP). 

The  FXXITP  was  established  in  1995  by  the  U.S.  Army  Training  and  Doctrine  Command 
(TRADOC)  and  is  overseen  by  TRADOC’s  Directorate  of  Training  and  Doctrine  Development 
(DTDD)  at  Fort  Knox,  Kentucky.  The  DTDD  has  worked  with  the  U.S.  Army  Research  Institute 
for  the  Behavioral  and  Social  Sciences  (ARI)  to  research  and  develop  numerous  training 
products  for  the  program.  The  majority  of  the  work  was  performed  at  the  Armored  Forces 
Research  Unit  at  Fort  Knox,  with  additional  work  completed  at  the  Infantry  Forces  Research 
Unit  at  Fort  Benning,  Georgia.  The  work  centered  on  the  development  of  simulation-based, 
structured  training  support  packages  (TSPs)  designed  for  the  collective  training  of  reserve 
component  (RC)  and  active  component  (AC)  battalion  and  brigade  staffs. 

Once  the  initial  FXXITP  products  had  reached  a  sufficient  state  of  maturity,  the  TRADOC 
Deputy  Chief  of  Staff  for  Training  (DCST)  directed  that  these  products  be  assessed  while  being 
used  by  AC  brigades.  The  assessment  was  to  focus  on  the  utility  of  the  TSPs  in  supporting  the 
brigades’  yearly  training,  specifically  in  preparation  for  National  Training  Center  rotations.  The 
ARI  was  to  provide  the  support  for  assessment  by  means  of  a  project  entitled  Implementation 
and  Support  for  the  Assessment  of  Force  XXI  Training  Program  (ISAT).  Along  with  a  team  of 
contractor  personnel,  ARI  conducted  the  assessment  with  the  assistance  of  four  AC  brigades 
between  April  1998  and  May  1999. 

While  the  primary  focus  of  the  DCST  directive  was  on  assessment  of  FXXITP  product 
utility,  ARI  directed  that  the  contractor  team  also  support  implementation  of  the  products,  to 
ensure  that  the  assessment  would  take  place  in  a  context  of  full  product  use.  Therefore,  the 
assessment  strategy  included  plans  for  a  support  infrastructure  that  would  facilitate  integration  of 
the  FXXITP  products  into  unit  training  plans. 

The  scope  of  the  assessment,  as  originally  planned,  was  adjusted  continually  over  the  course 
of  the  project  due  to  external  constraints  and  other  demands  on  the  activities  of  the  participating 
units.  As  a  result,  the  assessment  questions  could  not  be  completely  addressed.  This  report 
discusses  the  background  of  the  ISAT  project  and  documents  project  activities  and  outcomes. 

The  implementation  and  support  history,  assessment  results,  and  lessons  learned  were  briefed  to 
the  office  of  the  DCST  on  21  July  1999.  The  findings,  despite  the  compromises  to  scientific 


rigor,  should  support  the  development  and  fielding  of  TSPs  that  will  improve  the  near-term 
readiness  of  the  Army’s  AC  and  RC  forces.  Army  policy  makers  and  training  developers  will 
find  this  report  useful  in  continuing  their  efforts  to  provide  usable  and  affordable  training 
products  for  units. 


ZITA  M.  SIMUTIS 
Technical  Director 
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OPERATIONAL  ASSESSMENT  OF  FORCE  XXI  TRAINING  PRODUCTS: 
LESSONS  FOR  SUCCESSFUL  FIELDING 


EXECUTIVE  SUMMARY 


Research  Requirement: 

The  U.S.  Army  is  facing  the  challenge  of  meeting  complex  training  requirements  with 
increasingly  constrained  resources.  As  a  result,  Army  trainers  are  exploring  emerging 
technologies  and  innovative  techniques  that  can  be  used  to  achieve  training  goals.  Part  of  this 
exploration  has  been  accomplished  through  the  Force  XXI  Training  Program  (FXXITP). 

The  FXXITP  was  established  in  1995  by  the  U.S.  Army  Training  and  Doctrine  Command 
(TRADOC)  and  is  overseen  by  TRADOC’s  Directorate  of  Training  and  Doctrine  Development 
(DTDD)  at  Fort  Knox,  Kentucky.  The  DTDD  has  worked  with  the  U.S.  Army  Research  Institute 
for  the  Behavioral  and  Social  Sciences  (ARI)  to  research  and  develop  numerous  training 
products  for  the  program.  In  1998,  the  TRADOC  Deputy  Chief  of  Staff  for  Training  (DCST) 
directed  that  these  products  be  assessed  while  being  used  by  active  component  brigades.  While 
the  primary  focus  of  the  DCST  directive  was  on  assessment  of  FXXITP  product  utility,  ARI 
directed  that  the  contractor  team  also  support  implementation  of  the  products,  to  ensure  that  the 
assessment  would  take  place  in  a  context  of  full  product  use.  Therefore,  the  assessment  strategy 
included  plans  for  a  support  infrastructure  that  would  facilitate  integration  of  the  FXXITP 
products  into  unit  training  plans. 

The  project  that  was  organized  to  support  the  DCST  directive  was  titled  Implementation  and 
Support  for  the  Assessment  of  Force  XXI  Training  Program  (ISAT).  The  specific  objectives 
included:  (a)  to  develop  a  plan  to  guide  the  implementation  and  assessment  of  selected 
components  of  the  FXXITP;  (b)  to  build  an  implementation  support  infrastructure  at 
participating  unit  locations  to  facilitate  the  assessment;  (c)  to  conduct  the  implementation  and 
assessment  at  unit  locations  and  Fort  Irwin,  California,  (the  National  Training  Center  [NTC]); 
and  (d)  to  document  the  implementation  and  assessment  process,  the  results  of  the  assessment, 
required  changes  to  the  products,  and  suggestions  for  future  development. 

Procedure: 

The  project’s  Execution  Plan  (Human  Resources  Research  Organization  [HumRRO], 
Raytheon  Systems  Company,  TRW  S&ITG,  &  Litton  PRC,  1998c)  outlined  the  project’s 
strategy  for  accomplishing  the  objectives.  It  included  plans  for  managing  the  project  from  Fort 
Knox,  and  providing  support  to  the  two  selected  user  units.  Data  collection  for  the  assessment 
would  involve  interviews,  questionnaires,  and  observations,  as  outlined  in  the  Assessment  Plan 
(HumRRO  et  al.,  1998a). 

In  general,  the  Execution  Plan  outlined  four  activities  to  support  the  implementation: 

•  Purchase  hardware  (computers  and  associated  components)  to  support  use  of  Battle  Staff 
Training  System  (BSTS) 
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•  Reproduce  and  distribute  BSTS  software  and  CD-ROMs 

•  Reproduce  and  distribute  training  support  packages  (TSPs)  for  vignettes.  Brigade  Staff 

Exercise  (BSE)  and  Brigade  and  Battalion  Staff  Exercise  (BBSE) 

•  Establish  a  training  infrastructure  to  include 

-  on-site  personnel  at  the  user  unit  sites  who  would  provide  train-the-trainer  sessions 
on  all  FXXITP  products  and  assist  with  implementation  of  the  BSTS  and  vignettes, 

-  personnel  at  Fort  Benning  and  Fort  Knox  who  would  be  available  to  answer  difficult 
questions  about  the  FXXITP  products,  and 

-  provisions  for  a  surge  team  assembled  on  an  as-needed  basis  to  travel  to  the  user 
units  and  assist  with  implementation  of  the  BSE  and  BBSE. 

Similarly,  the  assessment  requirement  was  supported  by  four  activities: 

•  Observe  and  document  the  implementation  and  assessment  processes 

•  Conduct  data  collection  from  participants  by  means  of  interviews  and  questionnaires 

•  Conduct  quantitative  and  qualitative  analyses  of  the  assessment  data 

•  Synthesize  findings,  lessons  learned,  and  implications  for  FXXITP  future  directions. 

Because  only  a  limited  number  of  brigades  were  going  to  participate  in  the  implementation 
and  assessment,  and  because  there  was  no  opportunity  for  rigorous  control  of  the  brigade 
activities,  the  assessment  was  not  meant  to  be  a  full  evaluation  of  the  products’  value  or  impact. 
Instead,  ARI  and  DTDD  asked  for  a  complete  case  study  of  the  implementation  activities,  along 
with  intense  efforts  to  obtain  reactions  and  suggestions  from  all  those  involved  in  the  product 
use. 

Findings: 

The  ISAT  project  outcomes  represent  a  compilation  of  implementation  methods,  assessment 
data  and  analyses,  lessons,  and  recommendations,  summarized  in  this  report.  The  lessons 
address  considerations  of  acceptability,  impact,  supportability,  and  training  effectiveness 
assessment  itself.  Within  the  lessons,  the  research  proffers  solutions  to  the  identified  problems. 
Some  of  the  solutions  require  additional  research,  while  others  will  require  action  at  the  highest 
levels  of  Army  leadership.  Those  solutions  that  are  within  the  reach  of  training  designers  and 
developers  include  research  on  TSP  and  implementation  models,  redesign  of  TSP  products  and 
distribution  requirements,  and  planning  for  maintenance  and  sustainment  of  products.  Other 
solutions,  including  the  need  for  command  emphasis  at  division-level  and  higher,  and  the 
institutionalizing  of  the  products,  will  require  that  TRADOC  and  Army  leaders  make  a 
commitment  to  support  the  development  and  implementation  of  such  products  that  may  increase 
readiness  without  increasing  training  costs. 

Project  lessons  also  indicate  the  importance  of  maintaining  the  currency  of  training 
products,  and  of  providing  both  education  and  implementation  support  to  units  who  will  use  the 
products.  Additionally,  the  data  and  comments  from  users  highlight  the  importance  of  creating 
flexible  training  products  that  can  be  tailored  to  the  needs  of  the  user. 


Utilization  of  Findings: 

The  ISAT  project  has  generated  information  and  lessons  that  will  facilitate  the  fielding  of 
structured  training  products.  As  a  continuing  emphasis  is  placed  on  providing  low-resource, 
cost-effective  training  for  U.S.  Army  personnel,  this  report  can  lead  those  training  development 
efforts  into  the  selection  of  purposeful  design  and  implementation  initiatives. 
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OPERATIONAL  ASSESSMENT  OF  FORCE  XXI  TRAINING  PRODUCTS: 
LESSONS  FOR  SUCCESSFUL  FIELDING 


Introduction 

Since  1995,  the  Force  XXI  Training  Program  (FXXITP)  has  been  laying  the  groundwork  for 
a  systematic  program  of  structured  training  for  the  U.S.  Army.  The  research  and  development 
products  include  individual  training  on  staff  warfighting  skills  as  well  as  synchronization  skill 
exercises  at  the  collective  level.  The  methods  for  training  delivery  are  based  on  principles  of 
adult  training  and  learning,  and  incorporate  recent  technological  advances  in  computer-based 
instruction  and  simulation-based  training.  All  of  the  products  are  task-linked  to  doctrinal 
materials;  as  a  result,  a  logical  training  sequence  for  skill  acquisition  and  practice  is  inherent 
across  the  products,  and  all  products  can  be  identified  by  unit  trainers  as  supportive  of  their 
training  needs  and  objectives 

All  of  the  FXXITP  products  were  designed  and  developed  under  the  sponsorship  of  the 
FXXITP  and  the  U.S.  Army  Research  Institute  for  the  Behavioral  and  Social  Sciences  (ARI), 
following  the  time-honored  methods  of  Instructional  Systems  Design  (ISD),  as  explicated  in  the 
Army’s  Systems  Approach  to  Training  (SAT)  (Department  of  the  Army  [DA],  1995a).  During 
development,  each  product  was  formatively  evaluated  and  revised  several  times.  The  training 
materials  were  repeatedly  subjected  to  expert  reviews  by  representative  users  and  training 
developers.  The  completed  materials  are  constructed  as  exportable  training  support  packages 
(TSPs),  containing  virtually  everything  the  user  would  need  to  implement  the  products. 

Yet  despite  the  expertise  of  those  involved  in  the  design  and  development,  the  sound 
instructional  foundations,  and  the  intensive  formative  evaluation  processes,  none  of  the  products 
had  undergone  rigorous  summative  evaluation.  The  continuous  quality  control  activities 
embedded  in  the  design  and  development  process  ensured  that  the  materials  were  complete  and 
correct.  But  without  a  monitored  product  implementation  and  examination  of  the  effects  of 
product  use  on  staff  performance,  the  issue  of  training  effectiveness  was  still  open. 

Recognizing  this  important  gap  in  the  trail  of  evidence  for  product  acceptability,  the  Deputy 
Chief  of  Staff  for  Training  (DCST)  of  the  U.S.  Army  Training  and  Doctrine  Command 
(TRADOC)  directed  that  these  products  be  assessed  while  being  used  by  active  component  (AC) 
brigades.  The  assessment  was  to  focus  on  the  usefulness  of  the  TSPs  in  supporting  the  brigades’ 
yearly  training,  specifically  in  preparation  for  National  Training  Center  (NTC)  rotations. 

Working  with  the  FXXITP,  ARI  was  to  provide  the  support  for  assessment  by  means  of  a  project 
entitled  Implementation  and  Support  for  the  Assessment  of  Force  XXI  Training  Program  (ISAT). 
Along  with  a  team  of  contractor  personnel,  ARI  conducted  the  assessment  with  the  assistance  of 
four  brigades  between  April  1998  and  May  1999. 

While  the  primary  focus  of  the  DCST  directive  was  on  assessment  of  FXXITP  product 
utility,  ARI  directed  that  the  contractor  team  also  support  implementation  of  the  products,  to 
ensure  that  the  assessment  would  take  place  in  a  context  of  full  product  use.  Therefore,  the 
assessment  strategy  included  plans  for  a  support  infrastructure  that  would  facilitate  integration  of 
the  FXXITP  products  into  unit  training  plans. 
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This  report  describes  the  methods,  results,  and  conclusions  of  the  IS  AT  project’s  assessment 
of  the  FXXITP  products.  The  report  highlights: 

•  incorporation  of  the  FXXITP  products  into  the  units’  training  plans, 

•  assessment  procedures  and  results, 

•  recommended  changes  to  the  FXXITP  products,  and 

•  recommendations  for  future  training  programs. 

The  report  includes  six  sections: 

•  Section  1.  Project  Overview:  Describes  the  FXXITP  and  the  project’s  technical 
objectives  and  approach. 

•  Section  2.  Implementation  Plans  and  Reality:  Presents  the  project’s  implementation 
plan,  followed  by  a  discussion  of  how  units  actually  used  the  products. 

•  Section  3.  Assessment  Plans  and  Reality:  Presents  the  assessment  objectives  and 
planned  activities,  along  with  a  description  of  the  actual  data  collection  activities. 

•  Section  4.  Results  and  Discussion:  Presents  project  findings  related  to  the  critical  issues 
explored  during  the  assessment. 

•  Section  5.  Lessons  Learned:  Describes  the  implications  of  project  results  as  they  apply 
to  the  use  of  the  FXXITP  products  in  unit  training  and  FXXITP  product  design. 

•  Section  6.  Conclusions  and  Recommendations:  Discusses  the  value  of  the  FXXITP 
products,  factors  for  successful  implementation,  future  directions  for  the  FXXITP,  and  a 
strategy  for  fielding,  maintaining,  and  sustaining  the  FXXITP  products. 

Appendix  A  contains  a  list  of  acronyms  used  in  this  report. 

Section  1 .  Project  Overview 

At  the  start  of  the  ISAT  project,  ARI  outlined  the  project  requirements  in  terms  of  technical 
objectives  and  operational  tasks.  These  objectives  (described  below)  formed  the  basis  for  the 
project’s  implementation  plans,  including  its  general  assessment  approach  and  team 
organization. 

This  overview  begins  by  describing  the  FXXITP  products  that  had  been  selected  for 
assessment.  It  then  lists  the  project’s  objectives  and  describes  the  project’s  approach  to 
performing  the  stated  requirements  for  implementation  and  assessment. 


The  FXXITP  was  established  in  1995  by  TRADOC  and  is  overseen  by  TRADOC’s 
Directorate  of  Training  and  Doctrine  Development  (DTDD)  at  Fort  Knox,  Kentucky.  The 
DTDD  has  worked  with  ARI  to  research  and  develop  numerous  training  products  for  the 
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program.  The  FXXITP  was  designed  to  reduce  the  detrimental  impact  of  restrictions  on 
weapons  use  during  training,  terrain  limitations,  and  reduced  resources  available  for  attaining 
training  proficiency  and  readiness.  Several  products  have  been  developed  to  assist  units  by 
raising  their  level  of  task/skill  performance,  thereby  ensuring  more  effective  use  of  live  field¬ 
training  opportunities.  Four  of  the  FXXITP  training  products  were  selected  to  be  the  focus  of  the 
ISAT  implementation  and  assessment: 

•  Battle  Staff  Training  System  (BSTS), 

•  Brigade  Staff  Vignettes 1 , 

•  Brigade  Staff  Exercises  (BSE),  and 

•  Brigade  and  Battalion  Staff  Exercises  (BBSE). 

Research  and  development  on  the  BSTS  originated  at  ARI’s  Infantry  Forces  Research  Unit 
(Andre,  Wampler,  &  Olney,  1997)  and  continued  at  the  ARI  Armored  Forces  Research  Unit 
(AFRU).  Research  and  development  that  led  to  production  of  the  brigade  staff  vignettes,  BSE, 
and  BBSE  was  conducted  by  ARI  AFRU.  The  vignettes  and  the  BSE  were  completed  in  1996 
(C.  H.  Campbell,  Graves,  Deter,  &  Quinkert,  1998),  and  the  BBSE  was  completed  in  1998  (C.  H. 
Campbell  et  al.,  1999). 

The  Battle  Staff  Training  System 

The  BSTS  consists  of  functional  area  TSPs  for  individual  battalion-  and  brigade-level  staff 
officers.  The  TSPs  combine  computer-based  instruction  (CBI)  and  text.  Each  TSP  presents  a 
course  of  instruction  by  means  of  CD-ROM  based  products  and  supplemental  text-based 
instruction.  Additionally,  a  training  management  system  (TMS),  Environment  for  MultiMedia 
interactive  instruction  (EMMii),  allows  the  trainer  or  other  designated  individual  in  the  unit  to 
monitor  the  progress  of  individual  staff  officers  as  they  proceed  through  the  courses.  Courses 
train  commanders  and  staff  officers  in  their  individual  warfighting  skills  to  enhance  their 
proficiency  in  synchronization  of  battlefield  operating  systems. 

The  Brigade  Staff  Vignettes 

The  brigade  staff  vignettes  are  24  short,  self-contained  training  activities  that  focus  on 
specific  staff  process  events.  These  are  structured  training  events  in  that  they  provide  all  the 
necessary  components  to  implement  and  conduct  meaningful  training.  Each  vignette  is  designed 
to  provide  practice  and  feedback  on  explicit  objectives  and  tasks.  The  TSP  for  a  vignette  defines 
the  objectives,  outcomes,  and  limits  of  the  training  experience.  The  structure  also  includes  the 
tactical  scenario  that  provides  the  framework  for  the  required  activities.  Four  of  the  24  vignettes 
use  constructive  simulation  (Brigade/Battalion  Battle  Simulation  [BBS]  or  Janus)  to  generate 
scenario  events,  while  the  rest  use  live  simulation. 


1  The  vignettes  were  later  renamed  Staff  Group  Exercises  by  DTDD. 
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The  Brigade  Staff  Exercise 


The  BSE  components  include  integrated  scenarios  covering  the  preparation,  planning,  and 
execution  phases  of  the  battle.  The  BSE  scenarios  are  developed  for  constructive  simulation 
conducted  in  the  Mojave  Desert,  including  the  NTC  terrain  “box.”  The  TSP  for  the  exercise  also 
contains  brigade  staff  preparation  materials  and  guidance,  simulation  system  electronic  files  and 
documentation,  instructions  for  interactors  and  roleplayers,  materials  and  guidance  to  support 
observation  and  feedback,  and  complete  details  for  exercise  conduct  and  control.  The  majority 
of  the  work  provided  from  this  research  effort  focuses  on  the  brigade  commander  and  selected 
staff  members.  The  primary  training  audience  members  are  shown  in  Table  1. 


Table  1 

Primary  Training  Audience  for  the  Brigade  Staff  Exercise 


Training  Audience  Member 


Brigade  Commander 
Adjutant  (SI) 

Operations  Officer  (S3) 

Fire  Support  Officer  (FSO) 

Air  Defense  Artillery  Coordinator 
Army  Aviation  Liaison  Officer 
Military  Police  Platoon  Leader 
Forward  Support  Battalion  Commander 


Executive  Officer 
Intelligence  Officer  (S2) 

Supply/Logistics  Officer  (S4) 

Fire  Support  Coordinator 
Brigade  Engineer 
Chemical  Officer 
Signal  Officer 

Direct  Support  Military  Intelligence  Company 
Commander 


The  Brigade  and  Battalion  Staff  Exercise 

The  BBSE  is  a  structured,  multiechelon,  battlestaff  training  exercise  that  encompasses 
planning,  preparation,  execution,  and  sustainment.  It  allows  battalion  and  brigade  commanders 
and  staffs  to  interact  as  they  plan  for  tactical  missions,  employ  combat  power,  and  conduct  rear 
area  sustainment  operations.  A  training  scenario  generates  an  NTC-like  operational  tempo 
requiring  24-hour  operations  and  stresses  concurrent  and  parallel  planning  processes.  The 
exercise  allows  the  unit  to  practice  and  receive  feedback  on  key  staff  procedures,  which  enable 
the  unit  to  enter  major  field  training  exercises  at  a  higher  level  of  proficiency  in  the  staff  process. 
The  BBSE  generates  the  information,  cues,  and  simulated  operations  that  allow  combat  service 
support  (CSS)  to  be  a  major  consideration. 

The  BSE,  BBSE,  and  the  four  simulation-based  vignettes  require  the  support  of  an 
established  simulation  center  that  can  conduct  brigade-level  exercises  using  BBS  and  Janus.  The 
TSPs  contain  details  specific  to  use  of  simulation  in  these  particular  exercises. 

By  the  time  of  the  ISAT  project,  each  of  the  four  FXXITP  products  had  been  formatively 
evaluated  and  delivered  to  the  Army.  However,  analyses  to  determine  the  effectiveness  of  the 
products  upon  their  integration  into  unit  training  strategies  had  not  yet  been  conducted.  Thus, 
the  ISAT  assessment  focused  only  selectively  on  evaluating  TSP  designs,  and  more  on 
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evaluating  products  for  their  capacity  to  support  unit  training  plans.  The  objectives  and  tasks  of 
the  ISAT  project  were  designed  to  facilitate  such  an  assessment. 

Project  Objectives 

The  vision  of  the  project’s  procedures  and  outcomes  was  presented  by  ARI  in  terms  of  four 
technical  objectives: 

•  To  develop  a  plan  to  guide  the  implementation  and  assessment  of  selected  components 
of  the  FXXITP. 

•  To  build  an  implementation  support  infrastructure  at  participating  unit  locations  to 
facilitate  the  assessment. 

•  To  conduct  the  implementation  and  assessment  at  unit  locations  and  Fort  Irwin, 
California  (the  NTC). 

•  To  document  the  implementation  and  assessment  process,  the  results  of  the  assessment, 
required  changes  to  the  products,  and  suggestions  for  future  development. 

The  ISAT  team  developed  both  an  execution  plan  and  an  assessment  plan  (HumRRO  et  al., 
1998a,  1998c)  that  addressed  the  requirements  of  the  first  objective,  and  then  followed  the  plan 
to  accomplish  the  second  and  third  objectives.  The  fourth  objective,  documentation,  is  largely 
met  by  the  production  of  this  report. 

The  remaining  parts  of  this  section  briefly  describe  the  activities  outlined  in  the  execution 
plan.  Sections  2  and  3  will  provide  details  about  the  implementation  infrastructure  and 
assessment  plans. 


The  ISAT  Project  Execution  and  Assessment  Approach 

The  project’s  Execution  Plan  (HumRRO  et  al.,  1998c)  outlined  the  project’s  strategy  for 
accomplishing  the  objectives.  It  included  plans  for  managing  the  project  from  Fort  Knox  and 
providing  support  to  the  user  units  at  other  locations.  Data  collection  for  the  assessment  would 
involve  interviews,  questionnaires,  and  observations,  as  outlined  in  the  Assessment  Plan 
(HumRRO  et  al.,  1998a). 

In  general,  the  Execution  Plan  outlined  four  activities  to  support  the  implementation: 

•  Purchase  hardware  (personal  computers  and  associated  components)  to  support  use  of 
BSTS 

•  Reproduce  and  distribute  BSTS  software  and  CD-ROMs 

•  Reproduce  and  distribute  TSPs  for  vignettes,  BSE  and  BBSE 

•  Establish  a  training  infrastructure  to  include 

-  on-site  personnel  at  the  brigade  locations  who  would  provide  train-the-trainer  (T3) 
sessions  on  all  FXXITP  products,  assist  with  implementation  of  the  BSTS  and 
vignettes,  and  assist  units  in  understanding  and  taking  advantage  of  the  FXXITP 
opportunities; 
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-  personnel  at  Fort  Benning  and  Fort  Knox  who  would  be  available  to  answer  difficult 
questions  about  the  FXXITP  products;  and 

-  provisions  for  a  surge  team  assembled  on  an  as-needed  basis  to  travel  to  the  brigade 
locations  to  assist  with  T3  sessions  for  BSTS  and  vignettes  and  implementation  of 
the  BSE  and  BBSE. 

Similarly,  the  assessment  requirement  was  supported  by  four  activities: 

•  Observe  and  document  the  implementation  and  assessment  processes 

•  Conduct  data  collection  from  users  and  other  participants  by  means  of  interviews  and 
questionnaires 

•  Conduct  quantitative  and  qualitative  analyses  of  the  assessment  data 

•  Synthesize  findings,  lessons  learned,  and  implications  for  FXXITP  future  directions. 

Because  only  two  brigades  were  going  to  participate  in  the  implementation  and  assessment, 
and  because  there  was  no  opportunity  for  rigorous  control  of  the  brigade  activities,  the 
assessment  could  not  serve  as  a  full  evaluation  of  the  products’  value,  effectiveness,  or  impact. 
Instead,  ARI  and  DTDD  asked  for  program  monitoring  of  the  implementation  activities,  with 
intense  efforts  to  obtain  reactions  and  suggestions  from  all  those  involved  in  the  product  use. 

Details  of  the  plans  for  implementation  are  presented  in  Section  2  of  this  report,  along  with  a 
description  of  how  the  implementation  actually  occurred.  Section  3  similarly  describes  the 
assessment  approach  and  summarizes  the  extent  of  the  data  collection  over  the  course  of  the 
project. 


Section  2.  Implementation  Support:  Plans  and  Reality 

As  a  general  approach,  the  ISAT  project’s  assessment  focused  on  the  potential  success  of 
the  FXXITP  products  in  the  hands  of  the  users.  The  basic  strategy  was  to  monitor  users  while 
they  worked  with  the  training  products,  and  to  gather  feedback  regarding  suitability, 
acceptability,  training  effectiveness,  outcomes,  and  supportability.  Focusing  on  the  impact  of  the 
training  products  rather  than  on  their  quality  distinguished  the  ISAT  assessment  from  the 
established  formative  evaluation  approach  used  in  previous  training  development  research  (e.g., 
C.  H.  Campbell,  Deter,  &  Quinkert,  1997). 

Assessing  impact,  however,  demands  either  an  assumption  or  an  assurance  of  product  use. 
This  facet  of  assessment  is  often  referred  to  as  a  program  monitoring  or  as  an  accountability 
study  (King,  Morris,  &  Fitz-Gibbon,  1987;  Rossi  &  Freeman,  1993).  Accountability  studies  are 
frequently  the  result  of  external  mandates,  and  focus  on  one  or  more  aspects  of  the  ways  a 
program  is  being  implemented,  as  compared  to  its  design  characteristics.  Program  monitoring 
includes  systematic  examination  of  program  coverage  and  (particularly  relevant  here)  delivery. 

Because  all  of  the  FXXITP  products  were  newly  in  the  hands  of  users,  there  was  no  history 
of  use  and  no  established  infrastructure  for  user  support.  The  user  guidance  contained  in  the 
TSPs  had  been  judged  to  be  sufficient  by  instructional  experts,  and  the  products  had  been  used  in 
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trial  implementations.  But  even  in  those  trial  implementations,  the  development  team  was  on 
hand  to  assist  with  any  instances  of  confusion  or  inconsistency.  After  the  trials,  the  materials 
were  revised  to  eliminate  the  problems,  but  there  was  still  no  clear  evidence  that  the  products 
were  usable  by  units  without  further  mentoring  or  assistance.  Therefore,  ARI  directed  that  the 
ISAT  team  assist  as  necessary  to  ensure  that  the  products  could  be  used,  and  monitor  all  aspects 
of  the  attempted  implementation. 

A  distinctive  characteristic  of  this  project  was  that  the  participating  brigades  would  be 
preparing  for  a  rotation  at  the  NTC.  While  the  impending  NTC  rotation  greatly  increased  the 
potential  for  direct  feedback  on  the  effectiveness  of  the  training  as  implemented,  the  schedule  for 
such  a  train-up  is  intense.  As  a  result,  the  FXXITP  products  would  likely  compete  with  events 
from  existing  unit  strategies  routinely  used  to  prepare  for  NTC  rotations.  Within  the  ISAT 
project,  the  intent  was  to  work  with  the  units  and  provide  recommendations  that  would  allow  the 
products  to  complement  the  units’  training  strategies.  This  would  leverage  use  of  the  training 
products  to  better  prepare  the  units  to  adjust  to  the  faster  paced  training  environment  associated 
with  an  NTC  rotation.  The  ISAT  approach  emphasized  a  high  level  of  support  during 
preparation  and  conduct  of  training  exercises  to  increase  the  likelihood  that  units  would  be  able 
to  use,  and  would  use,  the  products  being  assessed. 

Despite  the  support  from  the  ISAT  team,  it  was  anticipated  that  use  of  the  products  could  be 
affected  by  outside  factors  such  as  operational  contingency  missions  (e.g.,  Intrinsic  Action,  Team 
Spirit),  weather  related  disasters,  and  shortfalls  of  training  funds.  These  conditions  could  cause 
the  units  to  alter  the  way  they  used  the  products.  The  accommodations  required  by  such 
disruptions  would  be  instructive  from  a  case  study  and  lessons  learned  perspective,  but  it  was 
acknowledged  that  they  could  also  affect  the  validity  of  the  program  assessment. 

As  events  transpired,  expectations  regarding  the  disruptive  effects  of  NTC  preparation  and 
other  contingencies  were  borne  out.  Even  as  the  ISAT  project  began,  unit  participation  was  in  a 
state  of  flux.  This  section  first  describes  the  effect  of  external  events  on  unit  participation,  and 
then  describes  the  implementation  support  plan  as  it  existed  when  unit  participation  had  been 
decided.  The  section  concludes  with  a  description  of  the  activities  and  milestones  in  supporting 
implementation  of  the  FXXITP  products  throughout  the  course  of  the  ISAT  project.  The 
description  further  details  the  influence  of  other  demands  on  unit  time  and  planning,  especially 
those  demands  placed  by  the  impending  NTC  rotation. 

In  describing  the  project’s  key  implementation  events,  this  section  focuses  on  the  use  of  the 
FXXITP  products  within  the  unit  training  schedules,  but  also  addresses  non-FXXITP  products 
and  training  events  that  had  effects  on  the  assessment  results.  In  addition  to  providing  a  history 
of  how  effectively  the  project’s  implementations  would  support  the  assessment  purposes  and 
goals,  the  content  of  this  section  will  set  the  stage  for  project  findings  and  results  presented  in 
Section  4. 


Unit  Participation 

As  the  ISAT  project  began,  TRADOC  DCST  coordinated  with  the  U.S.  Forces  Command 
(FORSCOM)  to  confirm  the  tasking  of  units  to  support  the  assessment.  The  TRADOC  DCST 
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had  directed  that  the  assessment  be  conducted  with  two  brigades  within  EH  Corps.  The  brigades 
were  to  be  preparing  for  their  NTC  rotations  as  this  would  best  accommodate  the  long-term 
ISAT  assessment.  One  of  the  two  brigades  initially  selected  asked  to  be  replaced,  citing  their 
then  advanced  preparation  status  for  their  NTC  rotation.  Another  brigade  then  agreed  to 
participate  upon  the  condition  that  they  would  assist  in  the  assessment  of  only  those  products  that 
would  meet  their  peculiar  training  needs.  These  products  included  the  BSTS  and  vignettes,  but 
not  the  BSE  and  BBSE. 

By  May  1998,  one  of  the  brigades  had  begun  to  use  the  FXXITP  materials,  as  described 
below.  At  that  time,  however,  they  were  told  to  prepare  for  deployment.  Once  this  occurred,  the 
brigade  could  no  longer  support  the  ISAT  effort.  In  July  1998,  yet  another  brigade  volunteered 
to  participate  in  the  assessment.  The  participating  units,  then,  included  two  brigades,  one  of 
which  would  not  be  using  the  BSE  or  the  BBSE.  A  limited  amount  of  use  has  also  occurred  in 
the  first,  deployed,  brigade. 

Throughout  this  report,  the  three  brigades  that  used  the  FXXITP  products  will  be  referred  to 
as: 

•  Brigade  A — initially  selected,  deployed  after  limited  use  of  FXXITP 

•  Brigade  B — could  only  use  BSTS  and  vignettes  in  support  of  its  mission  and  current 
training  requirements 

•  Brigade  C — volunteered  later,  used  most  of  the  products. 

Implementation  Support  Plan 

The  implementation  support  activities  were  scheduled  to  unfold  by  way  of  a  planning  and 
preparation  phase  followed  by  an  execution  phase.  The  timing  of  the  project  activities  would  be 
linked  closely  to  the  training  schedules  of  the  two  supporting  brigades.  The  schedules  of  the  two 
brigades  were  not  in  any  way  synchronized  and  each  required  a  tailored  schedule  of  events 
supporting  product  use. 

Planning  and  Preparation 

Reports  on  the  development  of  the  FXXITP  products  have  indicated  that  first-time,  and  even 
subsequent,  implementations  of  the  products  have  required  a  fair  amount  of  implementation 
support  from  personnel  familiar  with  the  products  (Andre  et  al.,  1997;  C.  H.  Campbell,  Graves, 
et  al.,  1998;  C.  H.  Campbell  et  al.,  1999;  Graves,  Campbell,  Deter,  &  Quinkert,  1997;  Hoffman, 
Graves,  Koger,  Flynn,  &  Sever,  1995).  Although  one  of  this  project’s  critical  assessment  issues 
involved  investigating  the  extent  to  which  units  could  use  the  products  without  external  support, 
it  was  evident  that  some  support  would  be  required  to  facilitate  the  assessment.  As  a  result,  ARI 
directed  that  extensive  planning  and  preparation  for  product  use  be  conducted.  During  planning 
and  preparation,  the  training  products  would  be  reproduced  and  distributed,  training  support 
personnel  would  begin  operations,  and  the  user  units  would  receive  information  and  assistance 
on  fitting  the  products  into  unit  training  plans. 
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Work  on  this  task  began  as  soon  as  the  project  officially  started.  The  first  activity  concerned 
procurement  of  hardware  to  support  use  of  the  BSTS.  The  computer  hardware  has  been  chosen 
to  conform  to  the  requirements  of  the  BSTS  application,  as  specified  by  BSTS  developers.  In 
accordance  with  ARI  requirements,  10  multi-media  computers  were  purchased  for  use  in  the  two 
brigades:  within  each  brigade,  one  would  be  dedicated  to  brigade  staff  use,  and  three  would  be 
for  the  battalion  staff.  Additionally,  two  computers  would  be  installed  in  the  installation 
simulation  center  at  one  location  for  use  by  other  units  located  there. 

A  master  copy  of  each  TSP  (printed  materials,  electronic  data  files,  map  overlays, 
simulation  initialization  files,  and  other  materials)  was  obtained  from  ARI  AFRU  as 
government-furnished  materials.  The  ISAT  team  reproduced  and  distributed  the  materials  in 
sufficient  quantity  to  support  the  three  participating  brigades  (the  two  initially  selected  and  the 
later  added  brigade)  and  one  installation  simulation  center  (see  Table  2). 


Table  2 

Training  Product  Components  and  Quantities 


Product/Component 

Brigade  A 

Brigade  B 

Brigade  C 

Sim  Center 

Total 

BSTS  Multi-Media  Computers3 

4 

0 

4 

2 

10 

BSTS-Brigade  Courseware6 

1  set 

1  set 

1  set 

1  set 

4  sets 

•  CD-ROMs,  Student  Workbooks, 
Scenario  Orders,  Maps,  Overlays 
BSTS-Battalion  Courseware0 

5  sets0 

3  sets 

6  sets0 

1  set 

15  sets 

•  CD-ROMs,  Student  Workbooks, 
Scenario  Orders,  Maps,  Overlays 

Vignettes 

•  Printed  Materials,  Scenario  Orders, 

1  set 

1  set 

1  set 

1  set 

4  sets 

Maps,  Overlays 
•  Simulation  Files 

- 

1  set 

- 

1  set 

2  sets 

BSE  TSP 

•  Printed  Materials,  Scenario  Orders, 

1  set 

1  set 

2  sets 

4  sets 

Maps,  Overlays 
•  Simulation  Files 

- 

- 

2  sets 

2  sets 

BBSE  TSP 

•  Printed  Materials,  Scenario  Orders, 

1  set 

1  set 

2  sets 

4  sets 

Maps,  Overlays 
•  Simulation  Files 

_ 

_ 

2  sets 

2  sets 

Note.  aEach  computer  to  include  required  software.  '’For  each  division,  three  extra  sets  of  common  core  courseware 
per  unit  set.  cThe  distribution  to  Brigade  C  was  adjusted  because  of  emerging  lessons  learned  from  Brigade  A  use 
(see  text).  BSTS  =  Battle  Staff  Training  System;  BSE  =  Brigade  Staff  Exercise;  TSP  =  training  support  package; 
BBSE  =  Brigade  and  Battalion  Staff  Exercise. 

After  the  first  two  distributions  of  materials  to  Brigades  A  and  B,  the  team  increased  the 
training  material  to  six  battalion  sets  with  the  three  additional  common  core  sets  and  five 
additional  TMS  modules  per  battalion  set.  All  text  material  was  provided  on  CD-ROMs.  Each 
battalion  set  included  three  additional  sets  of  comprehensive  assessment  components  (COMPS) 
per  battalion.  The  COMPS  overlays  were  embedded  in  Acrobat  Reader  on  CD-ROMs,  and 
paper  copies  of  the  overlays  were  also  provided. 
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At  the  same  time,  the  ISAT  infrastructure  to  support  product  use  was  being  established.  The 
IS  AT  project  team  organization  is  depicted  in  Figure  1.  The  team  included  training  developers 
and  military  subject  matter  experts  (SMEs)  who  had  developed  the  FXXITP.  The  team  was  led 
by  a  Project  Leader  who  ensured  that  the  project  met  expected  quality  standards  and  presented  a 
complete,  coherent,  and  integrated  assessment  and  results  package.  The  Project  Leader  was  the 
primary  point  of  contact  and  communication  channel  between  the  ISAT  project  team  and  the 
ARI  Contracting  Officer’s  Representative  (COR),  who  interfaced  with  the  DTDD. 


Figure  1.  ISAT  team  organization. 

The  ISAT  Project  Leader  oversaw  the  day-to-day  functioning  of  the  project  staff.  Duties 
included  coordinating  the  implementation  and  assessment  events,  preparing  intermittent  progress 
reports,  tracking  internal  project  scheduling  and  milestones,  and  continuous  coordination  with 
the  participating  units.  These  varied  responsibilities  and  tasks  allowed  for  a  well-orchestrated 
assessment  effort  under  constantly  changing  operating  conditions.  Substitutions  of  participating 
units  and  individual  unit  training  requirements  continually  challenged  the  project’s  ability  to 
conduct  the  assessment  in  accordance  with  the  project’s  assessment  objectives  (described  in 
Section  3  of  this  report). 
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Acting  under  the  Project  Leader,  an  Assessment  Coordinator  developed  the  project’s 
assessment  plan  and  ensured  that  all  evaluation  activities  were  carried  out  with  careful  attention 
to  detail.  The  Assessment  Coordinator  coordinated  the  data  collection  efforts,  analysis,  and 
documentation  during  product  implementation  at  the  brigade  locations  and  during  participating 
unit  NTC  rotations.  Both  the  Project  Leader  and  Assessment  Coordinator  were  located  at  Fort 
Knox,  but  traveled  to  implementation  sites  as  required. 

The  project  located  a  Training  Site  Team  on  site  for  Brigade  A  and  Brigade  B.  This  team 
included  a  Team  Chief,  two  Training  Support  Coordinators  (TSCs)  for  the  units  participating  in 
the  assessment,  and  a  TSC  for  the  simulation  center.  As  product  implementations  wound  down 
(October  1998),  the  team  at  that  site  was  reduced  to  one  person,  the  Team  Chief. 

The  Team  Chief  coordinated  and  directed  project  activities  at  the  site.  He  mentored  and 
assisted  the  units  while  he  oversaw  and  coordinated  the  training  support  for  the  exercises.  He 
also  assisted  with  the  initial  orientation  training  to  the  corps  and  division  leaders,  provided  the 
training  for  brigade  leaders,  and  directed  the  training  for  brigade  and  battalion  training  managers. 
He  was  directly  responsible  for  carrying  out  requirements  of  the  assessment  plan  at  that  location. 

The  TSCs  supported  and  coordinated  with  participating  brigades  and  the  simulation  centers. 
Between  them,  the  TSCs  represented  a  skill  mix  of  primary  expertise  in  BSTS,  the  vignettes  and 
the  BSE/BBSE,  and  the  BBS  and  Janus  simulation  systems.  The  TSCs  provided  training 
assistance  to  the  participating  units,  collected  assessment  data,  and  provided  problem-solving 
support  during  the  course  of  the  project.  These  TSCs  had  the  background  and  experience  to 
establish  credibility  with  the  training  unit  and  to  assess  the  training  products  in  a  professional 
manner. 

When  the  requirement  to  involve  Brigade  C  was  approved,  the  project  positioned  one  TSC 
there.  This  TSC  mentored  and  assisted  the  unit  while  he  oversaw  and  coordinated  training 
support.  He  assisted  with  orientation  training  to  the  installation  leaders,  provided  training  to  the 
brigade  leaders,  and  directed  the  training  for  brigade  and  battalion  training  managers.  He  was 
directly  responsible  for  carrying  out  requirements  of  the  assessment  plan  at  that  location. 

The  training  site  staff  members  were  thoroughly  familiar  with  each  of  the  Force  XXI 
products.  Additionally,  the  contractor  team  members  who  developed  the  products  were  available 
by  phone  and  e-mail  to  solve  problems  and  assist  as  required. 

During  the  planning  and  preparation  phase,  there  was  an  intense  proactive  involvement  with 
the  participating  brigades  throughout  the  home-station  training  cycle.  That  involvement  began 
with  a  thorough  informational  program  for  leaders,  training  managers,  training  support  personnel 
and  users  of  the  various  products.  This  orientation  was  considered  essential  to  proper 
implementation  of  the  packages  and  their  assessment.  With  support  and  general  guidance  from 
higher  echelons,  the  IS  AT  staff  then  worked  with  the  participating  brigade  commanders  and  staff 
to  integrate  the  products  into  their  long  range  training  calendars.  The  purpose  of  these  periods  of 
orientation  were  to  ensure  that  training  managers  understood  the  intent,  focus,  organization,  and 
support  requirements  associated  with  the  various  products  and  would  be  able  to  sequence  them  in 
a  manner  that  supported  the  unit’s  overall  training  strategy. 
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Also  during  the  initial  training  periods,  the  IS  AT  staff  briefed  the  simulation  center 
personnel  on  the  FXXITP  products.  The  simulation  center  personnel  were  trained  to  implement 
BSTS  and  monitor  vignettes  if  other  units  at  that  location  wanted  to  use  them.  Because  the  BSE 
and  BBSE  were  designed  for  brigade  and  battalion  exercises  based  on  corps  and  division  orders 
and  overviews,  members  of  the  corps  staff,  the  exercise  divisions  staffs,  and  the  simulation 
center  staffs  were  to  be  included  in  the  initial  training  effort. 

The  orientation  sessions  were  followed  by  T3  sessions,  provided  to  the  participating 
brigades  prior  to  their  initial  use  of  the  FXXITP  products.  The  T3  in  preparation  for  BSTS  and 
vignette  implementations  were  combined  into  a  single  session  and  comprised  three  days  of 
instruction.  The  first  day  of  this  T3  session  focused  on  BSTS  at  the  user  level.  During  the 
second  day,  further  training  on  BSTS  was  provided  to  unit  BSTS  administrators,  and  unit 
training  coordinators  were  instructed  on  the  usage  and  conduct  of  the  vignettes.  The  third  day 
focused  solely  on  vignette  training  for  the  unit  staffs. 

BSTS  training  covered  the  courses  available,  their  general  content,  time  requirements  and 
procedures  for  using  the  CBI  components,  and  other  topics  suggested  by  the  product  developers. 
The  training  stressed  the  requirements  for  recording  progress  and  for  completing  the 
comprehensive  assessments  as  laid  out  in  the  training  management  component.  Also  included 
was  the  requirement  to  provide  usage  data  and  feedback  on  user  reactions  as  part  of  the 
assessment  effort.  Training  included  sufficient  technical  information  to  permit  learning  center 
staffs  and  individual  users  to  properly  use  the  CBI  components. 

Training  on  the  brigade  staff  vignettes  was  designed  to  be  conducted  with  the  leaders  and 
staffs  of  the  participating  brigades.  The  training  included  an  orientation  on  the  materials  and 
how  they  are  organized  and  presented,  and  how  to  determine  reproduction,  distribution  and 
preparation  requirements  that  must  be  accomplished  prior  to  the  actual  training.  The  role  of  the 
TSC  in  assisting  the  unit  with  scheduling,  preparing  for  and  conducting  the  training  was  also 
stressed. 

The  T3  for  the  BSE  and  BBSE  was  modeled  after  the  T3  program  incorporated  in  the  BBSE 
TSP  itself.  It  was  preceded  by  informational  sessions  for  the  commanders  and  brigade  staffs  and 
the  battle  simulation  staffs.  The  information  covered  the  various  components  that  are  found  in 
the  TSP,  how  to  address  the  long  range  planning  requirements,  reproduction  of  the  guides  and 
TSP  components  that  would  be  used  during  a  particular  exercise,  preparation  steps  for  the 
simulation  center,  how  to  determine  staffing  requirements,  arranging  for  appropriate 
administrative  and  tactical  communications,  selecting  command  post  (CP)  arrangements,  and 
conduct  of  pre-exercise  training  and  preparations.  The  three-day  T3  session  itself  was 
decentralized  and  focused  on  the  following  groups: 

•  unit-provided  roleplayers  and  interactors  assigned  to  the  BBS  workstations, 

•  the  exercise  control  (EXCON)  cell,  which  roleplays  the  higher  and  adjacent  units, 

•  the  opposing  force  (OPFOR)  controller, 

•  the  observers  who  provide  feedback  to  the  training  audience,  and 

•  the  key  training  audience  personnel. 
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Unlike  the  BSTS  and  vignette  T3  sessions,  this  training  was  designed  to  be  conducted  by  the 
leaders  of  each  of  the  cells  and  the  simulation  center  staff,  who  worked  from  the  materials 
provided  in  the  TSP.  The  IS  AT  team’s  role  during  these  sessions  was  to  assist  the  trainers  with 
issues  and  address  concerns  with  the  TSPs. 

Execution  Phase 


During  the  execution  phase,  the  ISAT  team  supported  the  participating  brigades  using  a 
train-assist-train  approach  to  facilitate  the  learning  process  and  assist  the  units  in  mastering  the 
TSPs  and  training  material.  The  intent  was  to  allow  the  units  to  concentrate  on  the  staff 
procedures  and  the  military  decision-making  process  and  not  be  encumbered  by  administrative 
requirements.  While  an  important  aspect  of  the  assessment  was  “Can  the  unit  set  up  and  use  the 
FXXITP  products  without  assistance?”,  any  value  associated  with  the  products  could  be  masked 
if  the  units  were  overwhelmed  with  the  mechanics  of  setting  up  and  running  the  exercises. 
Therefore,  it  was  important  to  the  overall  assessment  to  assist  with  the  logistics  of  the  process 
and  allow  the  assessment  to  concentrate  on  the  products  themselves  and  the  contribution  they 
may  provide  to  the  overall  proficiency  of  the  staff.  To  the  extent  possible,  the  ISAT  team’s 
assistance  was  to  be  transparent  to  the  training  units.  The  team  would  support  the  units’  training 
and  assist  where  required  or  requested  on  a  non-interference  basis. 

Subsequent  to  the  training  described  above,  the  ISAT  team  continued  close  coordination 
with  and  support  of  the  participating  brigades  and  battalions  as  they  planned  for  and  used  the 
products.  The  on-site  TSCs  provided  immediate  solutions  for  almost  all  problems  related  to 
FXXITP  products.  If  the  on-site  TSCs  were  unable  to  resolve  a  problem,  they  consulted  with 
other  experts  at  Fort  Benning  and  Fort  Knox  in  order  to  provide  a  quick  response  to  a  support 
request. 

The  TSCs  provided  advice  and  assistance  to  the  staffs  of  the  units  in  using  the  TSPs  to  select 
and  implement  the  various  options  available  within  the  products.  They  answered  questions  to 
help  units  understand  the  rationale  for  particular  design  features  and  coached  staff  officers  and 
soldiers  who  were  assigned  responsibility  for  readying  TSP  components  for  particular  exercises. 
The  team  did  not  actually  make  the  copies  and  materials  which  were  used  during  conduct  of  the 
exercises.  This  remains  a  unit  responsibility  and  an  item  of  interest  in  the  assessment  of  the 
thoroughness  of  the  instructions  in  detailing  and  supporting  unit  preparation  requirements. 

Where  shortfalls  in  the  instructions  were  noted  or  the  unit  required  further  explanation,  the  team 
took  action  to  ensure  that  the  preparations  did  not  detract  from  the  quality  of  the  training. 

When  one  of  the  participating  brigades  prepared  to  conduct  a  BSE  or  BBSE,  the  TSCs 
guided  the  brigades  in  the  selection  of  missions  and  scenarios  to  be  trained  and  assist  with 
coordinating  preparation  for  the  event  with  the  simulation  center.  Just  before  the  BSE  or  BBSE 
was  actually  conducted,  a  surge  team  from  the  Fort  Knox  ISAT  staff  would  travel  to  the  brigade 
location  to  assist  with  the  event.  At  that  time,  they  monitored  and  assisted  with  training  for  a 
wide  range  of  participants  (e.g.,  observers,  roleplayers  portraying  enemy  and  friendly  units,  and 
simulation  controllers)  and  assisted  with  the  conduct  of  the  exercise. 
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The  IS  AT  team’s  support  provided  during  implementation  ranged  from  low-  to  high- 
intensity,  depending  on  the  product  being  implemented.  Low-intensity  support  was  provided 
during  BSTS  and  some  vignette  implementations,  and  high-intensity  support  was  provided 
during  any  simulation-based  training  event.  During  both  low-  and  high-intensity  events,  the 
assistance  focused  on  providing  advice  to  the  staffs  when  they  had  questions  or  concerns  with 
the  TSPs  and  answering  questions  about  the  rationale  for  a  particular  design  feature.  High- 
intensity  support  involved  the  presence  of  an  IS  AT  surge  team  and  often  required  that  the  team 
guide,  to  varying  extents,  the  course  and  conduct  of  the  exercises.  Consistent  with  the  project’s 
assessment  purposes,  all  support  was  documented  to  be  incorporated  in  assessment  findings 
regarding  how  well  and  easily  the  units  were  able  to  conduct  the  training. 

One  additional  duty  of  the  ISAT  team,  not  anticipated  in  the  execution  plan,  involved  rapid- 
response  training  development.  The  need  arose  as  a  result  of  a  requirement  for  the  simulation 
center  supporting  brigade  to  commit  their  BBS  systems  in  support  of  a  corps-level  ramp-up 
exercise.  The  simulation  center  could  not  conduct  the  scheduled  BSE  without  the  BBS  systems, 
and  subsequently,  the  unit  asked  the  ISAT  team  to  provide  a  replacement  exercise  that  would 
provide  benefits  similar  to  those  of  the  BSE.  The  development  effort  produced  a  brigade-level, 
Janus-based,  simulation  exercise. 


Implementation  Events 

As  a  general  rule,  the  units  implemented  the  FXXITP  training  products  in  progressive 
fashion,  beginning  at  the  individual  level.  Thus,  training  began  with  the  BSTS  products,  then 
progressed  to  the  vignettes  (with  some  overlap),  and  (for  Brigade  C)  culminated  with  the  BBSE 
just  prior  to  their  NTC  rotation.  The  units  interspersed  these  events  with  their  own  training 
exercises  as  they  executed  their  training  schedule  leading  up  to  the  NTC  rotation. 

Utilization  of  the  Battle  Staff  Training  System 

The  ISAT  team  supported  and  assessed  three  implementations  of  the  BSTS.  Distribution  of 
BSTS  TSP  materials  and  equipment  to  the  units  was  conducted  in  accordance  with  the  Modified 
ISAT  Execution  Plan  (HumRRO  et  al.,  1998d)  as  described  earlier  (Table  2).  Each 
implementation  was  different,  as  described  below. 

Brigade  A  established  a  brigade  learning  center  with  their  multi-media  computers  and  the 
TMS  computer  provided  by  the  project,  and  configured  the  BSTS  computers  in  a  local  area 
network  (LAN).  The  unit  also  allowed  the  installation  of  BSTS  software  on  unit-owned  and 
personal  computers  in  a  stand-alone  configuration.  When  the  brigade  deployed,  the  TMS 
computer  was  returned  to  the  ISAT  team.  Brigade  B,  which  was  not  provided  any  computers, 
installed  the  BSTS  into  unit-owned  and  personal  computers  in  a  stand-alone  mode.  They 
provided  usage  data  to  the  ISAT  team  who  managed  the  TMS  data  input  using  the  computer 
initially  provided  to  Brigade  A. 

The  two  computers  provided  to  the  simulation  center  were  configured  in  a  stand-alone 
mode.  Similarly  to  Brigade  A,  Brigade  C  established  a  brigade  learning  center  with  the  four 
multi-media  computers  and  the  TMS  computer  provided  by  the  project.  However,  they 
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configured  the  BSTS  computers  in  the  stand-alone  configuration  due  to  lessons  learned  in  earlier 
installations.  Brigade  C  also  allowed  the  installation  of  BSTS  software  on  unit-owned  and 
personal  computers  in  a  stand-alone  configuration. 

A  BSTS  T3  session  was  provided  to  Brigade  A  and  Brigade  C  prior  to  initial  use  of  the 
BSTS  courses.  For  both  participating  brigades,  ISAT  team  members  presented  BSTS  user  and 
trainer  orientation  sessions  at  the  training  site.  Brigade  B  elected  not  to  receive  formal  T3 
training,  stating  that  BSTS  was  supposed  to  be  exportable  and  should  not  require  T3  training. 

In  the  initial  implementation  in  Brigade  A,  the  brigade  and  task  force  staffs  chose  to  bypass 
the  BSTS  familiarization  instruction  provided  by  ISAT  surge  team  as  part  of  the  T3,  and  only  the 
brigade  staff  elected  to  receive  the  orientation  session.  As  a  result,  the  officers  began  with  an 
incomplete  understanding  of  the  system  and  many  of  them  encountered  technical  problems 
related  to  installing  and/or  operating  the  BSTS  software.  Especially  common  were  apparent 
system  failures  surrounding  the  manual  transfer  of  test  scores  from  stand-alone  computers  to  the 
TMS  mounted  on  a  central  server.  The  attendant  loss  of  data  in  the  TMS  gave  the  students  credit 
for  fewer  lessons  than  they  had  actually  completed.  These  “failures”  were  consistently  due  to 
users  following  improper  procedures,  but  nevertheless  generated  a  great  deal  of  frustration.  The 
on-site  Training  Site  Team  fielded  numerous  questions  and  problems,  provided  extensive 
diagnostic  and  trouble-shooting  services,  and  educated  the  users  about  system  capabilities  and 
limitations. 

At  both  brigade  and  battalion  echelons  within  Brigade  A,  the  participating  units  had 
insufficient  numbers  of  multimedia  computers  capable  of  running  BSTS  software.  Frequently 
this  led  officers  to  use  their  privately  owned  computers  in  order  to  meet  established  deadlines.  In 
this  assessment,  the  tasking  of  staff  officers  to  complete  their  BSTS  courses  was  by  necessity 
synchronized  very  tightly,  rather  than  distributed  over  time.  Additional  quantities  of  individual 
course  materials  (CD-ROMs,  workbooks,  maps,  overlays,  and  references)  were  distributed  to 
units  after  start-up  to  support  simultaneous  use  by  multiple  students. 

Because  of  limited  networking  capabilities  and  substantial  use  of  privately-owned 
computers,  many  individuals  used  BSTS  courseware  in  stand-alone  mode.  This  led  to 
bottlenecks  due  to  shortages  of  CD-ROMs,  especially  the  Brigade  and  Battalion  Common  Core 
course  CD-ROMs  and  EMMii  CD-ROMs  (normally  two  per  set).  Some  units  were  able  to  create 
their  own  copies  of  the  CD-ROMs.  The  stand-alone  environment  also  required  students  to 
upload  performance  data  to  the  TMS  manually  by  means  of  floppy  diskettes,  demanding 
additional  time  and  effort  and  engendering  resistance. 

Based  on  preliminary  feedback  from  the  BSTS  implementation  in  Brigade  A  and  Brigade  B, 
the  ISAT  team  was  able  to  make  several  improvements  as  they  prepared  for  implementation  for 
Brigade  C  (see  Table  3).  These  improvements  centered  on  expanding  the  quantities  of  key 
course  materials  and  making  tactical  graphics  readily  accessible.  Some  of  them  had  been 
implemented  informally  during  the  earlier  phase  of  the  assessment.  The  changes  had  a  positive 
influence  on  the  BSTS  implementation  environment  at  Brigade  C  and  the  quality  of  the 
subsequent  assessment  data. 
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Table  3 

Summary  of  Battle  Staff  Training  System  Distribution  Changes  for  Brigade  C 


Category  Brigade  Solution 

Problem 


Basis  of  Issue 

Issuing  one  brigade  set  and  three  battalion  sets 
to  a  brigade  created  bottlenecks  when  staff 
assistants  were  tasked  to  complete  courses. 

Issuing  one  EMMii  CD-ROM  with  each 
brigade  or  battalion  set  created  bottlenecks 
when  several  students  in  a  unit  tried  to  load 
software  on  unit-owned  or  student-owned 
computers  at  once. 

Issuing  one  brigade  and  one  battalion  COMPS 
CD-ROM  to  a  participating  brigade  created 
bottlenecks  when  several  students  were  ready 
to  take  the  COMPS  at  the  same  echelon. 

Accessibility  of  Overlays 

Providing  COMPS  overlays  only  in  Adobe 
Acrobat™  format  caused  delays  when  some 
students  had  to  locate  a  copy  of  Adobe  Acrobat 
Reader™. 

Printing  electronic  overlays  to  obtain 
hardcopies  led  to  “cut-and-paste”  assembly 
when  plotters  were  not  available. 


Issuing  six  battalion  sets  to  the  brigade 
enabled  all  maneuver  and  support  battalions 
to  work  on  courses  at  the  same  time. 

Issuing  three  EMMii  CD-ROMs  with  each 
brigade  or  battalion  set  enabled  more  staff 
officers  to  load  system  software  without 
delay. 

Issuing  three  COMPS  CD-ROMs  with  each 
set  of  courseware  enabled  more  students  to 
take  the  end-of-course  comprehensive  exam 
at  the  same  time. 

Providing  a  copy  of  Adobe  Acrobat 
Reader™  shareware  on  each  student  CD- 
ROM  enabled  immediate  viewing  of 
overlays. 

Including  a  full-size  paper  copy  of  each 
COMPS  overlay  in  each  set  of  courseware 
made  it  easier  for  unit  personnel  to 
reproduce  overlays. 


Note.  EMMii  =  Environment  for  MultiMedia  interactive  instruction;  COMPS  =  comprehensive  assessment 
component. 


In  preparation  for  implementation  within  Brigade  B,  the  higher-echelon  commander 
published  a  directive  that  all  brigade  and  maneuver  task  force  staff  officers  complete  their 
position-specific  courses  within  a  four-week  calendar  window.  In  response,  the  majority  of  the 
brigade  staff  and  part  of  one  task  force  staff  made  an  effort  to  devote  time  to  their  respective 
courses.  Because  the  directive  suggested  that  the  Brigade  and  Battalion  Common  Core  courses 
were  optional,  all  but  one  of  the  officers  began  with  their  appropriate  position-specific  course. 
Time  pressures  caused  many  of  the  students  to  use  a  shortcut  technique  that  involved  repeatedly 
taking  the  subject  pre-test  until  they  achieved  a  passing  score.  This  saved  them  the  time  of 
working  through  each  lesson  within  the  subject.  However,  it  subverted  the  course’s  training 
objectives  and  undoubtedly  imparted  a  distorted  view  of  the  value  of  the  training  products.  In 
addition,  lack  of  clear  expectations  about  the  end-of-course  COMPS  led  to  no  one  attempting 
this  component  of  BSTS.  The  combined  pressures  and  constraints  were  sufficiently  great  that 
fewer  than  half  of  the  target  audience  completed2  the  actual  lessons  within  the  designated  time 


2  In  light  of  the  approach  of  attacking  a  course  by  repeatedly  taking  the  pre-test,  “completion”  is  probably 
not  the  appropriate  term. 
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limit.  At  the  end  of  the  calendar  window,  competing  demands  forced  the  officers  to  turn  their 
attention  to  more  pressing  requirements. 

The  commander  of  Brigade  C  issued  instructions  that  all  brigade  headquarters,  maneuver 
task  forces,  and  support  battalion  staff  officers  should  complete  the  Common  Core  course  plus 
their  position-specific  courses,  with  no  time  limit  specified.  This  unit  followed  the  TSP- 
specified  execution  procedures  fairly  closely,  to  include  the  orientation  and  familiarization 
training  at  the  outset.  The  first  personnel  in  this  unit  began  their  course  work  immediately  after 
the  BSTS  T3  session,  approximately  two  months  prior  to  the  Leader  Training  Program  (LTP) 
visit  to  the  NTC,  and  BSTS  work  within  the  unit  continued  until  two  months  or  so  prior  to  the 
scheduled  NTC  rotation.  One  task  force  capitalized  on  a  one-week  Simulation  Networking 
(SIMNET)  training  visit  to  Fort  Knox  by  having  available  staff  officers  work  on  their  BSTS 
courses  between  collective  training  events.  One  staff  officer  in  this  group  chose  to  work  through 
his  course  with  several  of  his  staff  members  participating  throughout.  Another  group  also  used 
the  BSTS  courseware  for  staff  section  training  during  the  T3  period.  By  the  end  of  the 
assessment,  nearly  two-thirds  of  the  target  audience  had  completed  their  courses.  Unfortunately, 
the  manual  system  for  transferring  data  from  the  stand-alone  computers  to  the  TMS  captured 
only  partial  information  on  pre-test  and  post-test  exam  scores,  COMPS  scores,  and  time  taken  to 
complete  the  various  courses. 

The  task  force  that  spent  time  on  BSTS  course  work  during  a  Fort  Knox  visit  for  SIMNET 
training  provided  some  interesting  insights  into  the  time  required  to  complete  the  various 
courses.  Four  officers  were  able  to  complete  the  Battalion  Common  Core  Course  in  2-4  hours. 
Three  other  officers  completed  their  position-specific  courses  in  5-6  hours,  which  included  no 
reading  of  reference  materials.  These  times  were  substantially  less  than  those  indicated  in  the 
course  materials,  perhaps  reflecting  the  benefits  of  concentrating  study  efforts  in  a  one-day 
session.  This  finding  suggests  that  previous  estimates  of  required  time  may  be  high,  and  reliable 
data  in  the  operational  setting  might  be  worth  obtaining. 

Utilization  of  the  Vignettes 

Vignette  TSP  materials  were  distributed  to  each  of  the  brigades  and  the  simulation  centers 
as  described  earlier  and  shown  in  Table  2.  The  simulation  tapes  and  printed  materials  from  the 
support  coordinator  guides  for  the  simulation-based  vignettes  were  provided  to  the  simulation 
centers  because  they  are  responsible  for  providing  simulation  support  to  brigades  on  their 
installations.  The  brigades  were  expected  to  ensure  that  sufficient  copies  of  the  vignette 
materials  were  made  prior  to  the  execution  of  each  vignette. 

Vignette  T3  training  was  provided  to  Brigade  A  and  Brigade  C  during  the  same  week  that 
the  units  received  their  BSTS  T3  training.  For  both  of  these  brigades,  the  vignette  T3  training 
also  included  the  execution  of  one  vignette.  Brigade  B  did  not  receive  any  T3  training,  because 
they  wanted  to  validate  the  exportability  of  the  vignette  materials  as  they  did  with  the  BSTS 
materials.  Similar  to  their  experience  with  the  BSTS,  they  found  it  somewhat  difficult  to  execute 
the  vignettes  without  prior  training.  However,  part  of  the  problem  with  the  vignettes  was  caused 
because  they  made  significant  changes  to  the  vignettes  to  bring  the  scenarios  in  line  with  other 
training  they  were  involved  with  (described  below). 
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Brigade  B  conducted  three  vignettes  in  the  same  week,  as  part  of  a  staff  orders  drill  in 
preparation  for  an  upcoming  Army  experiment.  The  first  two  vignettes — Develop  a 
Reconnaissance  and  Surveillance  Plan,  and  Develop  a  Service  Support  Concept — were  done 
simultaneously,  with  no  direct  interaction  between  the  two  staff  groups  except  during  the  end-of- 
exercise  briefing  to  the  executive  officer  (XO).  The  brigade  staff  modified  the  vignette  materials 
by  using  the  tactical  materials  they  had  developed  for  the  experiment.  Little,  if  any, 
consideration  was  given  to  how  to  integrate  these  materials  into  the  vignettes,  and  how  to  use  the 
remaining  components  of  the  vignettes.  The  unit  also  altered  the  training  audience  substantially, 
expanding  the  participant  group  for  the  reconnaissance  and  surveillance  (R&S)  Plan  and 
substituting  the  Medical  Planning  Officer  for  the  S 1  in  the  Service  Support  Concept  vignette. 

The  third  vignette — Conduct  a  Brigade  Rehearsal — departed  from  the  TSP-specified  procedures 
as  well,  with  staff  captains  playing  the  roles  of  the  task  force  commanders  and  the  event  ending 
prematurely  because  of  administrative  considerations.  For  none  of  these  three  vignettes  did  the 
staff  hold  preparation  sessions,  a  situation  brief,  or  an  after  action  review  (AAR).  In  short,  the 
staff  of  this  unit  used  the  selected  vignettes  as  expedients  to  prepare  for  a  larger  simulation-based 
exercise,  and  they  followed  the  TSP-specified  procedures  very  loosely. 

Brigade  C  conducted  two  vignettes — Conduct  Mission  Analysis  and  Conduct  Course  of 
Action  (CO A)  Analysis — as  part  of  their  preparation  for  the  LTP.  Three  weeks  intervened 
between  the  two  events.  An  orientation  briefing  and  preparation  workshop  preceded  the  first 
vignette,  but  no  administrative  briefing  or  situation  brief  occurred  for  either  event.  The  brigade 
made  maximum  use  of  an  expanded  training  audience  that  included  most  assistant  staff  officers, 
although  no  staff  noncommissioned  officers  (NCOs)  were  present.  Without  an  administrative 
briefing,  there  was  confusion  among  some  “extra”  participants  as  to  the  nature  of  the  training. 
Several  of  these  participants  apparently  expected  they  were  attending  a  class  as  opposed  to  a 
practical  exercise.  No  Training  Coordinator  was  designated  for  either  of  these  vignettes,  so  no 
structured  observation  of  the  staff  processes  occurred  and  the  AAR  was  omitted.  The  staff 
appeared  to  focus  more  on  the  end  product  than  on  the  process  and  the  interaction  required  to 
achieve  the  product.  However,  the  XO  used  the  end-of-exercise  briefing  to  instruct  the 
participants  on  the  commander’s  expectations  and  critical  elements  of  information.  In  summary, 
the  staff  of  Brigade  C  used  two  vignettes  as  part  of  their  deliberate  strategy  to  prepare  for  the 
LTP.  While  they  did  not  follow  all  of  the  TSP-specified  procedures,  they  followed  them  more 
closely  than  did  their  counterparts  in  Brigade  B. 

The  vignettes  conducted  by  the  brigades  were  all  live  vignettes,  which  minimized  the 
support  requirements.  The  brigades  were  able  to  execute  each  of  the  vignettes  in  their  command 
posts  or  in  their  brigade  learning  centers.  The  biggest  support  problem  encountered  was  the 
inability  of  the  brigades  to  secure  the  Defense  Mapping  Agency  (DMA)-series  maps  required  for 
the  vignettes.  The  Fort  Knox  ISAT  team  provided  one  set  of  maps  needed  for  the  vignettes 
conducted  to  the  unit-located  ISAT  offices.  With  the  help  of  the  brigades,  additional  copies  of 
the  DMA-series  maps  were  made  to  meet  the  training  requirements  of  each  vignette  conducted. 

Utilization  of  the  Brigade  Staff  Exercise  and  the  Janus  Simulation  Exercise 

An  1 1th  hour  decision  to  commit  installation  BBS  assets  to  HI  Corps  for  a  high-priority 
exercise  left  Brigade  C  unable  to  conduct  the  BSE  as  planned.  In  lieu  of  the  BSE,  the  unit 


18 


implemented  a  “skeletal”  Janus  simulation  exercise  (SIMEX)  developed  on  short  notice  by  an 
ISAT  surge  team  working  at  Fort  Knox.  Centering  on  brigade  operations,  the  skeletal  TSP  was  a 
hybrid  package  resulting  from  the  merging  of  three  vignettes.  Because  of  the  lack  of  time  to 
develop  a  complete  TSP,  it  was  decided  that  an  ISAT  surge  team  would  fill  roles  as  EXCON  and 
OPFOR  support  for  the  exercise.  Janus  simulation  training  and  control  was  provided  by  staff 
from  the  Fort  Knox  Simulation  Center,  along  with  ISAT  simulations  experts.  As  a  result,  the 
team  designed  and  developed  only  those  TSP  components  needed  by  the  training  audience  (e.g., 
division  operation  order  [OPORD])  and  the  minimal  materials  that  the  surge  team  would  need. 

A  principal  component  of  the  “skeletal”  TSP  was  the  Performance  Objectives  (POs)  that 
came  from  the  BBSE  TSP,  used  to  facilitate  AAR  discussions  for  the  Janus  SIMEX.  As 
Brigade  C  prepared  for  the  brigade  operations-focused  Janus  SIMEX,  the  commander  selected 
five  PO  topics  to  serve  as  the  training  objectives  for  the  exercise: 

•  Reconnaissance  operations 

•  Parallel  planning 

•  Information  management 

•  Integration  of  fires 

•  Accelerated  decision-making. 

Thus,  the  Janus  SIMEX  provided  an  opportunity  to  assess  some  of  the  POs  as  well  as  the 
performance  contributions  of  the  training  already  accomplished  with  the  BSTS  courses  and 
vignettes. 

The  biggest  challenge  that  faced  the  surge  team  as  they  developed  the  TSP  was  providing 
read-ahead  materials  for  the  brigade  before  the  TSP  was  completed.  The  initial  thought  was  that 
the  Janus  SIMEX  would  be  a  mix  of  the  “Accelerated  Decision-Making  Process,”  “Conduct 
Parallel  Planning,”  and  the  “Prepare  and  Execute  a  Fragmentary  Order”  vignettes  and  that  the 
brigade  could  use  the  read-ahead  materials  from  these  vignettes  to  prepare  for  the  exercise.  Once 
the  team  started  to  combine  these  vignettes,  however,  they  realized  that  trying  to  pull  the  correct 
information  from  the  three  vignettes  was  too  confusing,  so  they  developed  a  separate  read-ahead 
packet  for  the  Janus  SIMEX.  However,  despite  efforts  to  eliminate  any  confusion  by  creating  a 
new  set  of  read-ahead  materials,  they  found  that  the  original  read-ahead  materials  for  the  three 
vignettes  were  also  distributed,  which  led  to  some  frustration  for  the  brigade  staff  members. 

For  the  Janus  SIMEX,  Brigade  C  executed  24  hours  of  mission  planning,  preparation,  and 
execution  across  two  days,  preceded  by  a  day  of  Janus  training  for  roleplayers  and  interactors  as 
well  as  setup  of  tactical  operations  center  (TOC)  and  communications  equipment.  The  ISAT 
effort  provided  a  10-person  surge  team,  supplemented  by  three  Janus  simulation  support 
personnel  from  the  Fort  Knox  Battle  Simulation  Center  (BSC).  The  surge  team  was  responsible 
for  controlling  the  exercise,  role-playing  the  higher  and  adjacent  headquarters  and  providing 
simulation  system  support. 

No  observers  supported  the  Janus  SIMEX  exercise,  contrary  to  the  TSP  instructions.  The 
training  audience  used  the  selected  POs  primarily  as  a  job  aid  or  reference,  focusing  more  on  a 
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monitor-plan-direct  process  briefed  by  the  commander  at  the  start  of  the  exercise.  Ad  hoc  AAR 
sessions  were  conducted  daily  by  the  XO,  who  also  led  a  capstone  AAR  on  the  last  day  of  the 
exercise.  The  surge  team  assisted  with  collection  of  AAR  data.  After  the  exercise  was 
completed  the  team  monitored  and  recorded  the  findings  during  the  brigade’s  final  AAR.  Two 
weeks  after  the  Janus  SIMEX,  the  surge  team  provided  the  brigade  with  a  tool  that  identified  the 
vignettes  and  BBSE  POs  that  would  help  them  correct  the  problems  noted  during  the  AAR. 

Utilization  of  the  Brigade  and  Battalion  Staff  Exercise 

BBSE  materials  were  provided  to  the  simulation  centers  between  March  and  July  of  1998  in 
accordance  with  the  modified  execution  plan  described  earlier  (HumRRO  et  al.,  1998d).  The 
simulation  tapes  and  supporting  simulation  files  provided  for  Brigade  C  were  later  updated  to  the 
latest  version  of  BBS  (version  5.1.1)  in  October  1998,  prior  to  the  conduct  of  the  BBSE  train-up 
in  December  1998  and  the  exercise  in  January  1999. 

Brigade  C  conducted  the  BBSE  mid-way  between  their  LTP  visit  and  their  NTC  rotation. 
The  commander  retained  the  same  five  POs  selected  for  the  Janus  SIMEX.  Because  of 
commitments  to  subsequent  training  activities,  the  unit  created  a  split  schedule  around  the 
Christmas  holidays.  Train-up  and  initial  mission  planning  for  the  BBSE  occurred  during  three 
days  before  the  holiday  period  began.  The  first  mission  execution  and  remainder  of  the  BBSE 
was  conducted  shortly  after  the  New  Year. 

Several  elements  external  to  Brigade  C  supported  the  preparation  and  execution  of  the 
BBSE.  The  division  headquarters  provided  the  Exercise  Director  and  personnel  to  staff  the 
division  response  cell  (EXCON).  A  sister  brigade  provided  the  observers,  including  the  Senior 
Observer/ Assistant  Exercise  Director,  and  a  control  cell,  including  the  Blue  Forces  (BLUFOR) 
Controller.  The  installation’s  simulation  center  provided  BBS  support  personnel  as  well  as 
coordination  and  preparation  support  for  the  exercise.  Members  of  the  ISAT  team  provided 
substantive  support  during  the  train-up,  but  served  only  as  data  collectors  and  advisors  during  the 
BBSE  itself. 

Prior  to  conduct  of  the  BBSE,  the  installation  held  three  separate  training  sessions,  for 
simulation  support  personnel,  EXCON  personnel,  and  observers.  The  BBS  simulation  train-up 
was  conducted  over  three  days  per  the  recommendation  of  the  BBSE  TSP.  The  EXCON  train-up 
lasted  about  four  hours,  mainly  consisting  of  an  organization  meeting  where  the  cell  leader  made 
assignments,  distributed  materials  and  established  a  time  for  the  key  personnel  to  meet  again. 
Ideally,  the  EXCON  cell  would  have  met  for  two  to  three  days  and  reviewed  the  first  and  second 
mission  orders  and  the  first  mission  key  messages.  The  observer  train-up  was  conducted  over  a 
two-day  period.  The  first  day  was  devoted  to  a  briefing  about  the  FXXITP  in  general  followed 
by  a  detailed  briefing  about  BBSE  performance  objectives  materials,  specifically,  the  five 
performance  objectives  being  used  for  the  January  BBSE.  The  ISAT  surge  team  provided  one 
person  to  each  training  session  for  support  and  observation. 

The  brigade  staff  started  planning  for  their  first  mission  while  the  train-up  was  underway. 
One  ISAT  surge  team  member  was  on  hand  to  observe  the  brigade’s  mission  planning  effort. 

The  main  exercise  itself  lasted  four  days,  during  which  the  unit  maintained  continuous  (24-hour) 
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operations.  On  the  first  day,  the  brigade,  maneuver  task  forces,  and  support  battalions  tactically 
deployed  their  respective  CPs  and  set  up  the  BBS  cells  while  the  simulation  center  personnel 
conducted  BBS  refresher  training  for  the  BBS  interactors  and  roleplayers.  The  observers/ 
controllers  evaluated  the  performance  of  brigade-  and  battalion-level  elements  and  conducted 
daily  AARs  and  NTC-like  change-of-mission  AARs.  The  Exercise  Director  conducted  twice- 
daily  exercise  control  meetings  to  coordinate  the  efforts  of  the  EXCON,  OPFOR,  simulation 
controllers,  and  observer/controllers  to  ensure  the  brigade’s  training  objectives  were  being  met. 
On  the  day  after  the  exercise  ended,  the  Exercise  Director  led  a  capstone  AAR. 

The  National  Training  Center  Leader  Training  Program 

The  FORSCOM  Regulation  350-50-1  (Department  of  the  Army  [DA],  1995b)  defines  the 
LTP  program  as  “a  command  and  staff  training  program  designed  to  prepare  a  unit  for  a 
rotation.”  Consistent  with  that  guidance,  the  Brigade  C  commander,  his  staff,  and  subordinate 
commanders  viewed  the  LTP  as  a  rehearsal  opportunity  in  preparation  for  their  April  NTC 
rotation.  Their  LTP  included  ground  reconnaissance  and  a  constructive  simulation  exercise  with 
AARs,  as  well  as  classroom  instruction. 

A  two-person  surge  team  followed  Brigade  C  to  Fort  Irwin  during  their  LTP  visit  in 
November  1998.  The  purpose  of  the  surge  team  trip  was  to  solicit  feedback  from  the  brigade  and 
LTP  staff  regarding  the  use  of  FXXITP  products  to  prepare  the  brigade  for  the  LTP  visit  and  to 
identify  additional  training  needs  of  the  brigade  that  can  be  supported  by  FXXITP  products. 
Additionally,  the  team  was  prepared  to  provide  the  LTP  and  NTC  staffs  with  a  briefing  to  give 
them  a  better  understanding  of  the  FXXITP  products  and  their  potential  use. 

The  National  Training  Center  Rotation 

The  Brigade  C  NTC  rotation  was  based  on  a  deployment  scenario,  and  began  with  one  week 
of  reception,  staging,  onward  movement,  and  integration  (RSOI)  training.  The  exercises  began 
with  force-on-force  missions  that  pitted  the  brigade,  as  a  divisional  supporting  effort,  against  the 
NTC  OPFOR.  The  first  mission  was  a  movement-to-contact,  followed  by  a  deliberate  attack,  a 
defense,  and  a  final  force-on-force  deliberate  attack.  Following  the  force-on-force  operation  the 
brigade  conducted  four  days  of  live  fire  operations.  Live  fire,  like  force-on-force,  included 
AARs  conducted  by  NTC  observers  for  the  brigade  and  its  subordinate  units.  Following  live 
fire,  the  brigade  conducted  recovery  operations. 

The  surge  team  for  the  NTC  rotation  included  six  members  of  the  project  staff.  Surge  team 
members  observed  most  aspects  of  the  brigade’s  NTC  experience,  including  the  RSOI  and  their 
24-hour  operations  on  all  three  missions.  The  surge  team  members  were  able  to  talk  with  the 
NTC  observers  and  the  brigade  staff  members  during  lulls  in  the  action. 

Summary 

This  section  has  presented  a  description  of  the  planned  implementation  support  activities  for 
the  FXXITP  products  within  two  brigades,  as  well  as  a  summary  of  the  actual  implementation 
that  took  place.  The  difference  between  the  plan  and  the  reality  is  important  in  interpreting  the 
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data  collection  results.  No  blame  can  be  attached  to  any  units  in  their  degree  of  participation,  but 
the  fact  remains  that  the  implementation  and  assessment  were  compromised  by  the  shortfall. 

Section  3.  Assessment  Methods:  Plans  and  Reality 

As  discussed  in  the  Introduction,  the  ISAT  project  was  a  response  to  a  TRADOC  DCST 
request  for  empirical  data  related  to  the  implementation  of  specific  FXXITP  products  among 
FORSCOM  brigades.  At  the  same  time,  the  project  represented  an  important  milestone  for 
ARI’s  training  development  efforts.  As  the  first  assessment  of  the  FXXITP  training  products  in 
a  routine  operating  environment,  the  effort  provided  an  opportunity  to  gather  valuable  data 
regarding  expected  performance  improvement,  TSP  usability,  value  realized  by  tactical  units, 
and  implementation  requirements.  The  assessment  was  to  serve  four  primary  purposes: 

•  Support  the  decision  needs  of  the  DCST  regarding  fielding  of  the  selected  training 
products. 

•  Support  the  limited  Training  Effectiveness  Analysis  being  conducted  by  the  DTDD, 

U.S.  Army  Armor  Center  (USAARMC). 

•  Provide  the  culminating  assessment  of  ARI  AFRU’s  research  and  development  efforts 
on  brigade  and  below  staff  training. 

•  Capture  information  about  how  the  FXXITP  products  should  be  modified  and  expanded 
for  future  use. 


Assessment  Plan 


The  Assessment  Plan  (HumRRO,  1998a)  outlined  a  series  of  assessment  activities  that 
supplemented  the  project’s  execution  plan  (HumRRO  et  al.,  1998c).  The  assessment  activities 
concentrated  on  determining  the  effectiveness  and  supportability  of  the  selected  products  as  they 
were  integrated  into  unit  training  programs  and  included  the  following: 

•  Collect  and  organize  data  related  to  the  training  effectiveness  and  implementation 
requirements  of  the  FXXITP  training  products. 

•  Analyze  and  interpret  the  accumulated  data  to  support  defensible  conclusions  and 
recommendations. 

•  Provide  an  audit  trail  documenting  assessment  methods,  including  units’  utilization  and 
management  of  the  TSP-supported  training. 

•  Identify  requirements  and  suggestions  for  improving  and  expanding  the  FXXITP 
training  products  in  support  of  future  training  needs,  especially  in  light  of  battlefield 
digitization. 

•  Capture  lessons  learned  regarding  implementation  and  utilization  of  the  training 
products. 


For  the  ISAT  assessment,  the  critical  issues  initially  took  the  form  of  five  levels  of  inquiry, 
with  each  level  building  on  the  preceding  level  in  a  progressive  fashion.  Four  of  these  levels 
originated  with  Kirkpatrick  (1994):  reaction,  learning,  behavior,  and  results.  The  fifth  issue — 
implementation — was  added  to  account  for  questions  relating  to  potential  fielding  decisions  and 
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future  development  efforts,  and  followed  the  model  of  a  program  monitoring  type  of  evaluation 
(Rossi  &  Freeman,  1993).  The  IS  AT  team  defined  each  level  in  terms  appropriate  to  the 
brigade’s  training  environment: 

•  Level  1,  Reaction — To  what  extent  does  the  training  audience  accept  and  value  the 
FXXITP  training  products? 

•  Level  2,  Learning — What  does  the  training  audience  leam,  with  reference  to 
achievement  of  training  objectives  specified  in  the  FXXITP  TSPs? 

•  Level  3,  Job  Performance — How  does  use  of  the  FXXITP  training  products  impact 
individual  and  collective  job  performance? 

•  Level  4,  Organizational  Impact — How  does  use  of  the  FXXITP  training  products  affect 
overall  achievement  of  unit  objectives  and  use  of  resources? 

•  Level  5,  Implementation — What  are  the  requirements  for  successful  fielding  of  the 
FXXITP  training  products? 

A  rigorous  evaluation  according  to  the  structure  of  the  four  Kirkpatrick  levels  (1994)  was 
known  at  the  outset  to  be  a  difficult  task.  Level  1  data  are  relatively  easy  to  collect,  consisting  of 
user  reactions  that  can  be  obtained  by  means  of  questionnaires  and  interviews,  and  Level  5  data 
can  be  derived  from  observations  as  well.  However,  the  Level  2,  Level  3,  and  Level  4 
evaluations  would  require  some  control  of  participant  activities,  and  would  be  strengthened  by 
use  of  control  groups.  Furthermore,  criterion  data  would  be  required  on  the  attainment  of 
training  objectives,  successful  job  performance,  and  institutional  impact.  Finally,  data  on  only 
two  or  three  brigades  would  not  constitute  particularly  strong  evidence  of  the  effects  of  the  use  of 
the  FXXITP  products. 

These  concerns  are  not  unique  to  the  ISAT  assessment,  nor  to  military  research  work.  Rossi 
and  Freeman  (1993)  stress  the  need  for  evaluators  to  be  responsive  to  the  context  in  which  they 
work,  especially  regarding  the  balance  between  the  press  for  evaluations  to  be  both  “scientific” 
and  “pragmatic.”  As  pointed  out  in  Raizen  and  Rossi  (1981),  monitoring  information  is  often  as 
important  as  information  on  a  program’s  impact.  Stake  (1979)  emphasizes  letting  evaluation 
emerge  from  program  observation,  and  would  encourage  letting  program  stakeholders  influence 
the  purpose  and  conduct  of  the  evaluation.  Even  within  the  constraints  of  the  formative- 
summative  evaluation  discussion,  evaluations  will  vary  according  to  their  purposes  (Shadish, 
Cook,  &  Leviton,  1991).  Thus,  from  the  outset,  the  ISAT  assessment  team  was  aware  of  the 
challenges  confronting  the  effort,  but  was  also  comforted  by  the  fact  that  the  wider  program 
evaluation  community  had  already  confronted  the  same  challenges  and  agreed  that  program 
monitoring  could  yield  information  as  useful  as  could  rigorous  impact  measurement. 

The  discussion  in  Patton  (1987)  led  the  ISAT  team  to  the  conclusion  that  a  qualitative 
approach  was  more  appropriate  in  the  assessment  of  FXXITP  product  implementation  and  use. 
Patton  states  that  “[qualitative  methods  permit  the  evaluator  to  study  selected  issues,  cases,  or 
events  in  depth  and  detail;  the  fact  that  data  collection  is  not  constrained  by  predetermined 
categories  of  analysis  constitutes  to  the  depth  and  detail  of  qualitative  data. . .  Qualitative  data 
provide  depth  and  detail  through  direct  quotation  and  careful  description  of  program  situations, 
events,  people,  interactions,  and  observed  behaviors”  (p.  9). 
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Therefore,  the  Kirkpatrick  levels  (1994)  were  used  to  organize  the  areas  of  inquiry,  but  were 
not  strictly  used  for  a  full  evaluation.  The  IS  AT  team  determined  that  data  collection  among 
training  audience  members  and  other  support  personnel  would  be  possible,  using  questionnaires 
and  interviews.  In  order  to  focus  the  questions,  the  team  used  the  issues  connected  with  the  four 
levels  to  focus  the  collection  of  participant  reactions.  The  data  would  address  perceptions  of 
learning,  job  performance,  and  organizational  impact  (Levels  2, 3,  and  4  topics),  but  would  not 
include  direct  or  objective  measures.  The  questionnaire  and  interview  data  would  be 
complemented  by  thorough  documentation  of  what  happened  and  what  could  have  been  done 
better.  In  this  way,  the  assessment  would  resemble  more  of  a  case  study  than  a  training 
evaluation. 

The  restructured  assessment  framework  aligned  the  original  five  factors — reaction,  learning, 
job  performance,  organizational  impact,  and  implementation — into  three  main  areas  of  interest: 

•  Acceptability  (incorporating  most  of  the  data  formerly  classified  under  reaction) — Were 
the  training  materials  doctrinally  correct?  Were  they  usable?  Were  other  materials 
necessary? 

•  Perceptions  of  Impact  (including  issues  of  learning,  job  performance,  and  organizational 
impact) — Were  the  training  products  useful  for  learning  and  practicing  job  requirements 
and  preparing  for  other  major  training  events?  Did  users  think  that  the  training  had  (or 
would  have)  a  positive  effect  on  performance? 

•  Supportability  (covering  the  questions  under  implementation) — What  would  it  take  to 
make  the  training  products  useable  within  a  brigade’s  training  plan? 

The  Program  Evaluation  Standards  (Joint  Committee  on  Standards  for  Educational 
Evaluation,  1994)  delineates  30  standards  categorized  in  four  groups  corresponding  to  four 
attributes  of  sound  and  fair  program  evaluation:  utility,  feasibility,  propriety,  and  accuracy. 
Utility  standards  guide  evaluations  so  that  they  will  be  informative,  timely,  and  influential,  and 
require  familiarity  with  the  evaluation  audience  and  its  information  needs.  Feasibility  standards 
help  to  ensure  that  the  evaluation  design  is  operable  in  a  field  setting  and  does  not  consume 
inordinate  resources.  Propriety  standards  promote  sensitivity  to  privacy,  freedom  of  information, 
and  the  protection  of  human  subjects.  Finally,  accuracy  standards  help  to  ensure  that  the 
evaluation  is  comprehensive,  the  obtained  information  is  technically  adequate,  and  the  judgments 
rendered  are  linked  logically  to  the  data.  These  were  the  standards  that  guided  the  ISAT 
assessment. 

Data  Requirements 

The  ISAT  team  began  the  process  of  identifying  data  requirements  by  developing  a 
comprehensive  list  of  questions  of  interest,  based  on  the  five  critical  issues  (which  were 
restructured  later  in  the  project).  To  ensure  that  the  issues  and  questions  important  to  the 
primary  stakeholders  were  addressed,  the  COR,  representatives  of  DTDD  and  the  USAARMC, 
and  the  developers  of  the  products  being  assessed  were  asked  to  contribute  questions  pertaining 
to  the  critical  issues  and  categories  of  information.  Table  4  lists  the  critical  issues,  along  with  the 
categories  of  information  forming  the  framework  for  questions  of  interest.  The  resulting  list  of 
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questions  of  interest  appears  in  Appendix  B.  The  questions  of  interest  shaped  the  development 
of  the  project’s  core  set  of  data  requirements. 

Table  4 

Framework  for  Questions  of  Interest 


Critical  Issue  Categories  of  Information 


Reaction-Acceptance  by  the  training 
audience 


Learning-Achievement  of  training 
objectives 

Job  Performance— Impact  on  individual  and 
collective  job  performance 


Organizational  Impact-Impact  on  overall 
achievement  of  organizational  objectives  and 
resources 

Implementation-Utilization, 
implementation  and  support  issues 


Credibility 
Ease  of  use 
Form  and  function 
Program  value 
Potential  enhancements 

Suitability  of  training  objectives 
Training  effectiveness 
Potential  enhancements 

Job  relevance 

Impact  on  staff  competence 
Impact  on  staff  performance 
Potential  enhancements 

Relevance  to  unit  goals  and  requirements 
Cost-benefit  considerations 
Potential  enhancements 

Training  management  and  product  utilization 
Compliance  with  training  support  package  procedures 
Support  requirements 
Implementation  considerations 
Lessons  learned 


Data  Collection  Instruments 


The  ISAT  assessment  was  designed  to  incorporate  a  wide  range  of  data  collection 
techniques — user  ratings,  written  and  oral  comments,  observations,  and  descriptive 
documentation.  In  addition  to  constructing  the  instruments  that  would  support  the  integration  of 
these  techniques,  the  ISAT  team  also  developed  instrument-specific  data  collection  procedures 
and  a  plan  for  analyzing  the  data. 

The  primary  data  collection  instruments  included  questionnaires,  structured  interview 
guides,  observation  guides,  a  significant  events  log,  and  a  lessons  learned  guide.  While  the 
questionnaires,  interview  guides,  and  observation  guides  were  tailored  to  each  product  and/or 
product  implementation,  the  events  log  and  lessons  learned  guide  were  designed  to  apply  across 
products  and  implementations.  Table  5  describes  briefly  each  type  of  instrument. 
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Table  5 

Types  of  Data  Collection  Instruments 


Data  Collection  Instrument 

Summary  Description 

Questionnaires 

Rating  scale  and  open-ended  items  designed  to  assess  the  success  of  each 
product  in  meeting  user  and  organizational  needs. 

Observation  Guides 

List  of  focus  issues  to  structure  and  refresh  monitoring  and  recording 
activities  of  on-site  observers. 

Structured  Interview  Guides 

Open-ended  questions  to  help  ISAT  interviewers  elicit  verbal  feedback 
on  the  major  products  and  their  training  value. 

Significant  Events  Log 

Checklist  and  open-ended  items  used  by  trained  observers  to  capture  and 
describe  significant  events. 

Lessons  Learned  Guide 

Instructions  and  checklist  used  by  ISAT  Team  members  to  capture 
insights  and  “next  time”  suggestions. 

For  the  assessment  of  each  FXXITP  product,  design  and  development  of  data  collection 
instruments  began  with  a  review  of  the  core  questions  of  interest  to  select  those  directly  relevant 
to  the  product  and  its  intended  utilization.  The  team  synthesized  the  questions  of  interest  to 
reduce  the  number,  and  then  transformed  the  questions  of  interest  into  individual  items  (e.g., 
rating  scale  questions  and  open-ended  items  for  questionnaires,  oral  questions  for  interviews). 
Quantitative  measurement  techniques  were  emphasized  in  the  construction  of  questionnaires.  To 
reduce  the  burden  placed  on  respondents,  questionnaires  and  interviews  were  designed  around  a 
20-minute  and  one-hour  time  limit,  respectively3. 

The  sources  of  information  for  the  questionnaires  and  interviews  varied  depending  on  the 
particular  data  collection  event.  Specific  versions  of  the  instruments  were  prepared  for  the 
training  audience  members,  support  personnel,  and  trainers  (such  as  observer/controllers, 
training  managers,  and  exercise  directors). 

This  general  method  produced  a  comprehensive  package  of  paper-based  instruments  for 
each  training  event  in  which  the  FXXITP  products  were  used  during  the  project.  In  addition, 
special  instruments  were  developed  for  the  NTC  events — the  LTP  and  the  brigade  rotation — and 
the  Janus  simulation  training  exercise  that  employed  only  parts  of  established  FXXITP  products 
(described  in  Section  2).  Appendix  C  contains  sample  data  collection  instruments,  illustrating 
the  approach  for  questionnaires,  interview  forms,  and  observation  guides. 

Data  Collection  Procedures 


A  principal  goal  of  the  assessment  was  to  collect  data  at  every  opportunity  at  which  the 
participating  units  used  the  various  training  products.  Throughout  the  course  of  the  ISAT 


3  All  instruments  used  outside  the  ISAT  Team  were  reviewed  by  the  U.S.  Army  Research  Institute  (ARI) 
Survey  Office  and  assigned  personnel  test  numbers. 
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project,  there  were  six  major  events  that  were  expected  to  provide  opportunities  for  data 
collection: 

•  BSTS  use 

•  Vignette  use 

•  Conduct  of  the  BSE 

•  NTC  LTP 

•  Conduct  of  the  BBSE 

•  NTC  Rotation. 

The  majority  of  the  data  collection  would  take  place  in  the  home  station  environment,  as  the 
units  used  the  various  FXXITP  products.  In  addition  to  the  home  station  events,  on-site  data 
collection  would  occur  during  scheduled  training  exercises  at  the  NTC,  to  gather  information 
about  the  impact  of  the  FXXITP  training  products  on  unit  performance  in  an  external  evaluation 
environment.  The  NTC  events  included  the  LTP  and  scheduled  unit  rotations.  A  portion  of  the 
data  from  NTC  events  would  be  collected  after  the  unit  returned  to  home  station.  Throughout 
the  assessment,  early  coordination  with  key  leaders  from  the  participating  brigades  would  be 
emphasized  to  optimize  the  data  collection  opportunities. 

The  ISAT  data  collection  involved  use  of  the  approved  instruments  designed  for  each  of  the 
data  sources,  following  the  data  collection  scheme  shown  in  Figure  2.  The  matrix  portrays  the 


Data  Sources 

Data  Collection  Events 

BSTS 

Courses 

Vignettes 

BSE 

LTP 

BBSE 

NTC 

Rotation 

Training  Audience 

V 

V 

Y 

V 

✓ 

V 

Support  Personnel 

V 

✓ 

Trainers 

V 

S 

S 

V 

V 

Exercise  Control  Personnel 

V 

Simulation  Center  Staff 

V 

ISAT  Observers 

y 

V 

V 

V 

V 

After  Action  Review  Packages 

S 

Figure  2.  Planned  sources  of  data  and  major  assessment  events. 

focus  of  the  data  collection  activities  for  each  of  the  products  and  events  of  interest.  For  each 
training  product,  multiple  data  collection  instruments  typically  were  used  to  cover  each  of  the 
data  sources. 

The  ISAT  team  members  were  to  serve  as  on-site  data  collectors  in  both  the  home  station 
and  NTC  arenas.  As  the  units  planned  and  prepared  for  using  the  products,  the  on-site  TSCs  not 
only  assisted,  but  also  recorded  questions,  comments,  and  observations  about  product  use.  The 
TSC  familiarity  with  the  units’  training  situation,  coupled  with  both  military  and  FXXITP 
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expertise,  would  make  it  possible  to  obtain  clear  and  relevant  insights  concerning  product  use. 
During  major  collective  training  events,  the  on-site  support  cell  was  augmented  by  a  surge  team. 
The  surge  team  data  collectors  were  selected  on  the  basis  of  active  duty  experience  with  mounted 
operations  as  well  as  familiarity  with  the  FXXITP  products.  The  surge  team  membership 
remained  relatively  stable  across  successive  training  events. 

The  data  collectors  received  training  by  the  IS  AT  Assessment  Coordinator  to:  (a)  explain 
the  objectives  of  the  assessment,  (b)  cover  basic  data  collection  assignments  and  scheduling,  and 
(c)  review  the  techniques  for  collecting  observational,  interview,  and  questionnaire  data.  The 
training  emphasized  the  importance  of  remaining  objective  so  that  personal  biases  regarding  the 
training  products  would  not  influence  the  collection  of  data. 

The  basic  data  collection  procedures  emphasized  minimal  intrusion  on  unit  training 
activities,  full  participation  of  users  in  planned  questionnaire  and  interview  sessions,  and  prompt 
completion  of  data  collection  requirements.  The  on-site  data  collectors  recorded  their 
observations,  administered  questionnaires,  conducted  interviews,  and  gather  data  from 
supplemental  sources.  To  record  their  observations,  the  collectors  used  observation  guides  and 
significant  event  logs  to  record  factual  information  about  events  as  well  as  their  own 
observations  and  judgments.  When  conducting  group  interviews,  the  data  collectors  operated  in 
two-person  teams  whenever  possible,  with  one  member  taking  notes  of  the  discussion.  Audio 
recordings  backed  up  the  note  taking. 

The  ISAT  team  members  planned  to  gather  lessons  learned  information  throughout  the 
entire  assessment,  using  special  guides  to  capture  a  broad  range  of  inputs.  The  lessons  learned 
guides  structured  the  capture  process  by  prompting  specific  observations,  judgments,  and 
insights  regarding  each  of  the  products.  Inputs  came  from  on-site  observations,  questionnaires, 
interview  records,  and  significant  event  logs. 

An  additional  occasion  for  obtaining  observations  about  the  FXXITP  products  came  when 
the  ISAT  team  was  invited  to  brief  the  four  products  (BSTS,  vignettes,  BSE,  and  BBSE)  for 
personnel  from  the  7  Army  Training  Center  (ATC).  The  briefing  included  information  sessions 
on  each  product,  walkthroughs  of  the  TSPs,  and  demonstrations  of  the  training  exercises.  The 
feedback  from  the  audience  was  not  analyzed  with  the  more  systematically  collected  data,  but  is 
discussed  in  Sections  5  and  6. 

Assessment  Events  and  Data  Collection 


As  described  in  Section  2,  there  were  many  departures  from  the  implementation  plans 
within  both  of  the  participating  brigades.  As  a  result,  the  data  collection  was  less  comprehensive 
than  had  been  anticipated,  and  results  were  not  easily  interpretable.  However,  these  difficulties 
had  been  anticipated  and  discussed  among  DTDD,  ARI,  and  ISAT  team  researchers,  and  various 
corrective  actions  were  taken  in  order  to  glean  the  greatest  amount  of  usable  information  for  the 
three  areas  of  interest.  Some  of  the  adjustments  were  documented  in  the  Modified  Assessment 
Plan  (HumRRO,  1998b),  although  other  changes  were  necessitated  throughout  the  project. 
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Because  of  the  wide  gap  between  the  BSTS  procedures  specified  in  the  FXXITP  TSP  and 
those  actually  used  in  the  initial  implementation,  the  questionnaire  data  from  the  Brigade  A  are 
not  included  in  this  report.  At  the  same  time,  the  extensive  input  regarding  improvement  of  the 
BSTS  products  and  the  valuable  lessons  learned  while  working  with  this  unit  are  fully 
represented  in  the  following  section. 


Table  6  provides  summary  information  about  the  amount  of  questionnaire  and  interview 
data  obtained  during  the  assessment.  Due  to  competing  requirements  beyond  the  units’  control, 
only  one  of  the  participating  brigades  was  able  to  afford  the  ISAT  team  a  complete  range  of  data 
collection  events  from  BSTS  coursework  through  the  unit’s  NTC  rotation.  Thus,  the  bulk  of  the 
data  came  from  a  single  FORSCOM  unit  representing  the  Army’s  active  duty  brigades.  Even 
within  that  one  brigade,  the  return  rate  for  questionnaires  was  very  low  despite  intense  efforts  by 
ISAT  TSCs  and  surge  team  members. 


Table  6 

Response  Rate  for  ISAT  Data  Collections 


Event 

Brigade  B 

Brigade  C 

Participating 

Providing  Data 

Participating 

Providing  Data 

BSTSa 

21 

11 

58 

7 

Vignettes6 

19c 

2 

42c 

17 

Janus  SIMEXd 

— 

— 

61c 

28 

BBSE6 

— 

— 

133 c 

34 

LTP 

— 

— 

41° 

17 

NTC  Rotation 

\ t  .  i  _  ar~> _ nn'T'O 

ur\ _ v  .  ,  »  .  t  i 

— 

1 15c 

13 

or  courses.  bFor  vignettes,  “Participants”  includes  training  audience  members.  'Approximate;  numbers  varied 
depending  on  the  phase  of  the  event.  dFor  the  Janus  SIMEX,  “Participants”  includes  training  audience  members, 
roleplayers,  and  simulation  controllers.  Tor  the  BBSE,  “Participants”  includes  training  audience  members  and 
Observer/Controllers.  BSTS  =  Battle  Staff  Training  System;  SIMEX  =  simulation  exercise;  BBSE  =  Brigade  and 
Battalion  Staff  Exercise;  LTP  =  leader  training  program. 


Data  Reduction  and  Analysis  Procedures 

In  preparation  for  the  data  reduction  and  analysis,  a  database  was  constructed  that 
accommodated  the  majority  of  the  collected  data.  A  family  of  files  was  created  to  correspond  to 
the  individual  questionnaires.  Only  numerical  data  were  entered  into  the  database  fields,  with 
the  aid  of  data  entry  screens  constructed  to  facilitate  keyboard  entry  directly  from  the 
questionnaires.  Text  fields  and  transcribed  interview  comments  were  entered  into  text  files.  The 
database  served  to  generate  organized  data  displays  for  inspection  and  to  support  descriptive 
statistical  analysis  of  quantitative  data. 

As  the  data  were  collected  from  the  training  sites,  the  ISAT  team  entered  them  into  the 
computer  database.  One  team  member  entered  the  data  directly  from  the  completed 
questionnaires  into  the  files,  using  data  entry  screens  and  keyboard.  Two  levels  of  quality 
control  (QC)  procedures  were  applied  to  the  database  contents.  First-level  QC  was  accomplished 
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by  the  data  entry  person.  Second-level  QC  activities  were  performed  by  the  database  manager, 
who  spot-checked  database  files  and  provided  feedback  to  the  data  entry  person. 

The  ISAT  team  analyzed  the  assessment  data  in  two  stages:  quick-look  processing  of  data 
as  they  became  available,  and  comprehensive  analysis  and  interpretation  of  cumulative  data. 

The  goal  of  this  strategy  was  to  identify  key  findings  in  a  timely  manner  while  reserving 
systematic  conclusions  for  thorough  analysis.  Each  of  these  stages  entailed  three  steps: 

•  organization  and  reduction  of  data, 

•  qualitative  and  quantitative  analysis,  and 

•  interpretation  of  data. 

Quick-look  analyses.  During  the  quick-look  analyses,  which  occurred  throughout  the 
project,  qualitative  data  were  organized  to  facilitate  rapid  identification  of  high-impact  findings. 
Issues  of  special  interest  were  generated  during  conversations  with  ARI  and  DTDD,  or  were 
formulated  as  a  result  of  significant  events  during  the  data  collection,  and  raw  data  (e.g., 
interview  notes,  completed  questionnaires)  were  assembled  and  collated  that  addressed  the 
issues.  In  general,  this  process  was  organized  to  highlight  recurring  observations  and  potential 
contributions  to  fielding  decisions.  Qualitative  techniques  were  then  used  to  illuminate  critical 
events,  trends,  and  patterns.  This  process  relied  on  the  judgment  of  the  team  members  to  identify 
and  enlighten  noteworthy  incidents  and  observations  of  immediate  interest,  with  operational 
factors  and  implications  in  mind. 

Comprehensive  analyses.  In  preparation  for  the  comprehensive  analysis,  the  ISAT  team 
segregated  the  data  into  qualitative  and  quantitative  components.  Qualitative  data  (comments, 
suggestions,  etc.)  were  organized  according  to  context  and  utility,  for  review  and  interpretation. 
For  these  purposes,  the  qualitative  data  items  were  tagged  to  indicate  their  source  (e.g.,  brigade 
S3)  and  then  sorted  under  a  set  of  categories  driven  by  the  nature  of  the  questions  that  elicited  the 
responses.  Quantitative  measures  were  entered  into  data  analysis  files  using  the  Statistical 
Package  for  the  Social  Sciences  (SPSS).  Data  in  the  SPSS  files  generally  required  no 
computation  or  other  reduction.  The  organization  built  into  the  database  files  mirrored  the 
structure  of  the  parent  questionnaires. 

For  the  comprehensive  analysis,  the  team  used  a  combination  of  qualitative  and  quantitative 
techniques.  The  ISAT  SMEs  and  Assessment  Coordinator  studied  the  qualitative  data  to 
determine  their  value  in  answering  the  questions  of  interest.  Quantitative  data  such  as  ratings 
and  frequency  counts  were  analyzed  using  descriptive  statistics  (e.g.,  means,  standard  deviations) 
when  the  sample  sizes  were  sufficient.  In  addition,  informal  comparisons  were  performed 
between  NTC  performance  of  the  participating  units  and  at-large  performance  trends  published 
by  the  Center  for  Army  Lessons  Learned  (CALL).  The  small  sample  sizes  and  the  lack  of  a 
control  group  precluded  the  use  of  most  parametric  and  nonparametric  statistical  techniques 
(e.g.,  chi  square,  regression  analysis)  to  analyze  the  quantitative  data. 
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Summary 


This  section  has  described  the  purposes,  objectives,  and  methods  of  the  IS  AT  project’s 
assessment  of  the  FXXITP  products.  The  assessment,  focusing  on  product  impact,  was  intended 
to  be  conducted  in  a  routine  operating  environment  and,  thus,  relied  on  the  integration  of  the 
FXXITP  products  into  unit  training  plans.  The  plans  were  modified  continually,  as  necessitated 
by  the  circumstances  of  unit  participation  and  resource  constraints.  The  substitution  of  one 
brigade  for  another  during  the  course  of  the  project  was  only  moderately  disruptive.  The  effects 
on  product  implementation  of  competing  demands  on  unit  time,  resources,  and  focus  were  a 
greater  confounding  issue. 


Section  4.  Results  and  Discussion 

This  chapter  presents  the  major  findings  of  the  project’s  assessment  and  discusses  their 
meaning.  For  reasons  discussed  in  Sections  2  and  3,  the  bulk  of  the  data  came  from  Brigade  C. 

In  addition  to  data  gathered  during  exercises  using  the  FXXITP  TSPs,  this  presentation 
summarizes  data  obtained  during  non-FXXITP  product  training  events  (i.e.,  the  Janus  SIMEX, 
LTP,  and  NTC  rotation)  and  FXXITP  product  reviews.  The  combination  of  sources  provides 
multi-faceted  information  regarding  the  acceptability,  impact,  and  supportability  of  the  FXXITP 
products.  As  discussed  in  Section  3,  those  three  areas  address  different  aspects  of  the  products’ 
quality  and  utility: 

•  Acceptability — Were  the  training  materials  doctrinally  correct?  Were  they  usable? 

Were  other  materials  necessary? 

•  Perceptions  of  Impact — Were  the  training  products  useful  for  learning  and  practicing  job 
requirements  and  preparing  for  other  major  training  events?  Did  users  think  that  the 
training  had  (or  would  have)  a  positive  effect  on  performance? 

•  Supportability — What  would  it  take  to  make  the  training  products  useable  within  a 
brigade’s  training  plan? 

Both  qualitative  and  quantitative  results  are  discussed.4  Because  of  the  small  sample  sizes, 
analysis  of  quantitative  data  is  limited  to  descriptive  techniques.  As  a  general  rule,  this  section 
will  present  the  data  for  each  of  the  products  in  a  standard  sequence,  beginning  with  acceptability 
and  ending  with  supportability.  Data  from  the  non-FXXITP  events  will  be  incorporated 
wherever  they  shed  light  on  a  particular  product.  The  section  concludes  with  results  relating  to 
the  assessment  of  the  FXXITP  as  a  whole,  and  how  the  individual  products  combined  to  produce 
effects  on  LTP  and  NTC  performance  and  on  the  unit  training  program.  Table  7  identifies  the 
events  that  contributed  to  the  assessment  of  each  of  the  FXXITP  products. 


4  Compiled  quantitative  data  from  the  questionnaires  completed  during  the  assessment,  write-in 
comments  from  the  same  questionnaires,  and  interview  transcripts  have  been  provided  to  U.S.  Army 
Research  Institute  (ARI).  The  quantitative  data  are  in  SPSS  files,  while  the  questionnaire  comments  and 
interview  transcripts  are  in  MS  Word®  files. 
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Table  7 

Force  XXI  Training  Program  Products  and  Assessment  Events 


Fvpnk  rVmtrihiitino  Data 

Products 

Relevant  to  Products 

BSTS 

Vignettes 

Janus  SIMEX 

BBSE 

BSTS  implementations 

X 

Vignette  implementations 

X 

X 

Janus  SIMEX 

X 

X 

X 

LTP 

X 

X 

X 

BBSE 

X 

X 

X 

X 

NTC  rotation 

X 

X 

X 

X 

Note.  BSTS  =  Battle  Staff  Training  System;  SIMEX  =  simulation  exercise;  BBSE  =  Brigade  and  Battalion  Staff 
Exercise;  LTP  =  leader  training  program. 


Interpretation  of  assessment  data  relied  heavily  on  close  collaboration  between  the  ISAT 
team’s  assessment  personnel  and  SMEs.  The  SMEs  provided  important  input  to  ensure  realistic 
consideration  of  operational  and  tactical  factors.  The  team  used  audit  trail  data,  including 
information  captured  in  the  significant  events  logs,  to  shed  light  on  trends  of  special  interest, 
with  emphasis  on  the  units’  implementation  and  management  procedures. 

The  turnover  of  key  personnel  on  the  various  brigade  and  battalion  staffs  was  a  confounding 
factor.  A  few  staff  officers  participating  in  NTC  training  had  not  used  the  FXXITP  products,  and 
equally  few  who  used  the  products  were  absent  for  NTC  exercises.  However,  not  all  eligible 
personnel  provided  data  during  any  given  data  collection  opportunity. 

One  concern  was  the  possible  influence  of  simply  involving  the  participants  in  an 
“assessment.”  Equipping  the  units  with  new  training  tools  and  signaling  expectations  of 
improved  performance  could,  by  itself,  lead  to  faulty  evidence  of  positive  effects — especially 
among  subjective  data.  This  phenomenon — known  as  the  “Hawthorne  effect” — is  discussed  by 
Cook  and  Campbell  (1979),  who  state  that  there  is  not  a  great  deal  of  evidence  of  the  effect 
occurring  in  field  experiments.  In  the  absence  of  a  control  condition,  the  team  relied  primarily 
on  patterns  of  indicators  (convergence  and  divergence  among  related  measures)  to  gauge  the  true 
contributions  of  the  FXXITP  products. 

Battle  Staff  Training  System 

Personnel  from  all  three  brigades  conducted  BSTS  courses  to  assist  in  the  project’s 
assessment.  Due  to  the  abbreviated  participation  of  Brigade  A  and  the  unique  implementation  by 
Brigade  B,  however,  the  assessment  described  presently  is  based  primarily  on  the  experiences 
and  feedback  from  Brigade  C.  The  assessment  presents  data  from  course  participants  and  also 
from  reviews  from  7ATC.  As  the  BSTS  is  designed  for  use  early  in  a  brigade’s  or  battalion’s 
training  cycle,  subsequent  training  events  (e.g.,  vignettes,  LTP,  NTC)  served  as  vehicles  to 
gather  information  regarding  the  more  extended  effects  of  BSTS  at  its  integration  into  a  training 
plan. 
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Battle  Staff  Training  System  Acceptability 


Acceptability  data  were  gathered  following  BSTS  implementations  from  September  and 
December  of  1998  and  during  the  7ATC  review  of  selected  courses.  Both  brigade  and  task  force 
users  from  Brigade  C  provided  feedback  via  questionnaires  and  interviews  on  the  currency  of  the 
doctrine  taught  and  the  capacity  of  the  TSP  to  support  implementation.  The  numbers  of 
individuals  responding  were  very  small,  leading  us  to  discuss  the  findings  in  very  general  terms 
of  trends  and  indicators. 

Table  8  shows  the  breakout  of  the  feedback  on  OPORDS,  graphic  overlays,  and  practical 
exercise  materials  from  the  few  Brigade  C  respondents.  A  majority  of  the  participants  (67%  - 
80%)  indicated  that  the  course  materials  were  consistent  with  current  doctrine.  A  key  comment 
from  the  7ATC  reviewers  during  their  visit  was  that  the  course  material,  while  doctrinally  sound, 
did  not  reflect  a  Division  XXI  structure  along  with  its  updated  doctrine  and  digitization.  Based 
on  this  feedback,  it  is  realistic  to  conclude  that  the  BSTS  is  sufficiently  acceptable  doctrinally, 
but  that  it  must  still  be  updated  so  that  future  commanders  can  keep  their  units  up  to  speed  on 
new  and  continually  evolving  doctrine. 


Table  8 

Perceptions  of  Doctrinal  Currency  of  Battle  Staff  Training  System  Materials 


Material 

Inconsistent  or 
Totally  Inconsistent 

Neither 

Consistent  or 
Totally  Consistent 

N 

Operation  orders 

0 

4  (80%) 

5 

Graphic  overlays 

1  (20%) 

1  (20%) 

3  (60%) 

5 

Practical  exercise  materials 

2  (33%) 

4  (67%) 

6 

The  capacity  of  the  TSP  to  support  training  was  assessed  by  examining  how  easily 
participants  were  able  to  use  the  materials,  how  well  the  course  materials  supported  the 
conditions  for  successful  training,  and  whether  or  not  the  amount  of  time  spent  on  the  training 
was  acceptable.  As  shown  in  Table  9,  participants  were  generally  able  to  find  information  in  the 
TSP.  Of  the  7  participants,  6  agreed  (to  a  moderate,  great,  or  very  great  extent)  that  the  course 
materials  established  the  conditions  necessary  for  successful  training.  Only  3  of  7  participants 
indicated  that  the  amount  of  time  required  by  the  training  was  acceptable;  the  other  4  participants 
were  split — 2  thought  the  amount  of  time  was  unacceptable  and  2  were  neutral. 
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Table  9 

Perceptions  of  Usefulness  Battle  Staff  Training  System  Materials 


Questionnaire  Item 

How  easy  to  find  information 

Difficult  or 
Very  Difficult 

Neither 

Easy  or 

Very  Easy 

N 

..  .in  student  guides 

0 

1  (17%) 

5  (83%) 

6 

. .  .in  computer-based  modules 

1  (14%) 

1  (14%) 

5  (71%) 

7 

No  Extent  or 
Slight  Extent 

Moderate 

Extent 

Great  Extent  or 
Very  Great  Extent 

N 

Extent  to  which  materials  establish 
conditions  necessary  for  successful  training 

1  (14%) 

2  (28%) 

4  (57%) 

7 

Not 

Acceptable 

Neutral 

Acceptable 

N 

Acceptability  of  time  required  by  the 
training 

2  (28%) 

2  (28%) 

3  (43%) 

7 

Table  10  summarizes  major  recommendations  for  enhancing  the  BSTS.  Many  of  the  items 
pertain  primarily  to  use  of  the  courseware  in  a  stand-alone  EMMii  environment.  If  a  new 
programming  environment  were  adopted,  recommendations  specific  to  EMMii  (e.g.,  transfer  and 
recovery  of  data  in  stand-alone  mode,  pre-test  management)  would  become  irrelevant.  Further, 
if  the  library  were  transitioned  to  an  Intranet/Intemet  environment,  some  of  the  program  start-up 
recommendations  would  be  unneeded. 


Table  10 

Summary  of  Recommended  Battle  Staff  Training  System  Training  Support  Package 
Enhancements 


Category 

Problem 

Recommendation 

Printed  Materials 

Critical  information  is  difficult  to  find  in  the  student 
instructions  of  the  course  workbook  . 

Providing  workbook  materials  in  only  hard-copy  form 
increases  the  risk  of  loss  of  materials  or  assembly 
errors  during  duplication. 

Electronic  files  of  workbook  text  and  job  aids  are  in  a 
format  inconsistent  with  current  Army  standards. 

TSP  contains  no  materials  to  help  trainers  diagnose  and 
solve  technical  problems. 

Redesign  the  student  workbook  for  ease 
of  use  (e.g.,  detailed  table  of  contents). 
Place  master  copy  of  all  workbook 
materials  (text,  job  aids,  and  graphics) 
on  user-friendly  CD-ROM. 

Convert  workbook  text  and  job  aid  files 
to  Army  standard  format. 

Add  a  troubleshooting  guide  for  the 
system  administrator  to  the  TSP. 

table  continues 
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Table  10  (continued) _ 

Category 

_ Problem  Recommendation 

Instructional  Software 

Critical  student  instructions  appear  only  in  printed 
materials  or  read-me  files,  making  it  easy  for  a  student 
to  miss  them. 

Current  software  does  not  limit  the  number  of  times  a 
subject  pre-test  can  be  taken,  allowing  a  student  to 
bypass  the  training  objectives. 

Mixing  paper  and  CBI  materials  breaks  up  the  flow  of 
lessons  and  complicates  the  instructional  process. 

Out-of-date  information  draws  criticism  from  the 
training  audience  and  degrades  the  face  validity  of  the 
courses. 

Relying  on  CD-ROM  technology  requires  multiple  sets 

_ for  each  unit  and  makes  updating  cumbersome. _ 

Note.  TSP  =  training  support  package;  CBI  =  computer-based  instruction. 


Place  instructions  in  a  CBI  module  to  be 
completed  before  lessons  can  begin. 

Add  help  files. 

Modify  student  software  to  prevent 
taking  a  subject  pre-test  more  than  once. 

Convert  entire  set  of  courses  to  100% 
CBI,  eliminating  the  course  workbooks. 
Establish  a  continuing  mechanism  for 
updating  doctrine,  terminology,  and 
symbols  reflected  in  the  TSPs. 

Migrate  the  entire  course  library  to  an 
Intranet/Intemet  environment. 


Battle  Staff  Training  System  Impact 

Impact  data  were  concerned  with  measuring  the  effects  of  BSTS  on  individual  and  unit 
performance.  The  data  were  gathered  following  BSTS  implementations  as  well  as  during 
subsequent  training  events  (i.e.,  vignettes,  LTP,  BBSE,  and  NTC  rotation).  The  feedback  on 
impact  represents  measures  of  perceived  or  estimated  impact,  rather  than  objectively  determined 
effects  of  the  training. 

Following  implementations  of  the  BSTS  courses,  users  were  asked  four  questions  regarding 
the  benefits  of  BSTS  to  the  individual  user.  When  asked  to  what  extent  the  course  allowed  for 
the  practice  of  battle  techniques,  4  of  7  respondents  were  positive.  On  the  second  item,  5  of  7 
respondents  said  that  the  BSTS  only  slightly  improved  their  understanding  of  their  tasks;  there 
were  no  respondents  who  indicated  that  their  task  understanding  had  not  improved  at  all. 
Similarly,  user  understanding  of  the  staff  was  reported  to  be  improved  to  a  slight  or  moderate 
extent  by  all  seven  respondents.  Improvements  in  task  performance  were  estimated  to  be  slight 
to  moderate,  as  three  respondents  indicated  they  improved  a  slight  extent,  three  to  a  moderate 
extent,  and  only  one  to  no  extent  at  all.  Results  are  presented  in  Table  11. 


Table  11 

Perceptions  of  the  Impact  of  Battle  Staff  Training  System  on  Individual  Users 


Area  of  Impact 

No  Extent  or 
Slight  Extent 

Moderate 

Extent 

Great  Extent  or 
Very  Great  Extent 

N 

Enabled  practice  on  battle  techniques 

3  (43%) 

2  (29%) 

2  (29%) 

7 

Improved  understanding  of  tasks 

5(71%) 

1  (14%) 

1  (14%) 

7 

Improved  understanding  of  the  staff 

3  (43%) 

4  (57%) 

0 

7 

Improved  task  performance 

4  (57%) 

3  (43%) 

0 

7 
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Given  frequent  user  comments  about  the  “refresher”  role  of  BSTS  course  work,  the  training 
audience  may  have  distinguished  between  acquiring  genuinely  new  knowledge  and  reinforcing 
knowledge  they  had  acquired  earlier.  For  example,  one  battalion  commander  successfully 
passed  every  subject  pre-test  without  taking  any  lessons,  but  he  stated  the  various  exams 
allowed  me  to  self-assess  areas  where  I  need  to  do  some  homework.”  When  it  came  to  the 
ability  to  perform  individual  tasks,  the  training  audience  may  have  reasoned  that  their 
performance  as  a  staff  officer  is  too  dependent  on  group  interaction  to  benefit  directly  from 
individual  self-study. 

Another  aspect  of  impact  reflected  the  need  to  determine  the  product’s  potential  impact  on 
unit  performance.  As  seen  in  Table  12,  most  BSTS  training  audience  respondents  judged  that 
the  course  offerings  contributed  to  their  unit’s  training  program  to  a  moderate  extent  or  greater 
(86%)  and  enhanced  the  unit’s  combat  readiness  (71%).  More  than  70%  estimated  the  courses 
saved  unit  training  time  and  funds. 


Table  12 

Perceptions  of  the  Impact  of  Battle  Staff  Training  System  on  the  Unit 


Area  of  Impact 

No  Extent  or 
Slight  Extent 

Moderate 

Extent 

Great  Extent  or 
Very  Great  Extent 

N 

Help  meet  staff  training  goals 

1  (14%) 

3  (43%) 

3  (43%) 

7 

Enhance  unit  combat  readiness 

2  (28%) 

3  (43%) 

2  (28%) 

7 

Save  unit  training  time 

1  (14%) 

4  (57%) 

2  (28%) 

7 

Save  unit  training  funds 

2  (28%) 

3  (43%) 

2  (28%) 

7 

Finally,  respondents  were  asked  to  rate  the  utility  of  the  BSTS  by  indicating  the  extent  to 
which  the  benefits  of  the  courses  outweighed  the  costs,  and  the  extent  to  which  their  own  brigade 
as  well  as  other  brigades  should  use  BSTS.  On  all  three  items,  results  were  positive,  as  shown  in 
Table  13.  Seventy-one  percent  of  the  respondents  said  that  the  benefits  outweighed  the  costs  to  a 
great  or  very  great  extent.  More  than  70%  agreed  that  the  BSTS  should  be  used  in  the  future  by 
their  own  unit  and  other  units. 


Table  13 

Perceptions  of  the  Future  Utility  of  the  Battle  Staff  Training  System 


Measure 

No  Extent  or 
Slight  Extent 

Moderate 

Extent 

Great  or  Very 
Great  Extent 

N 

Benefits  outweigh  costs 

1  (14%) 

1  (14%) 

5  (71%) 

7 

Own  brigade  or  task  force  should  use 

1  (14%) 

0 

6  (86%) 

7 

Other  brigades  or  task  forces  should  use 

1  (14%) 

1  (14%) 

5  (71%) 

7 

Impact  data  collected  following  implementation  were  supplemented  with  data  collected  as 
the  unit  proceeded  through  its  yearly  training  plan.  Following  critical  events  in  their  plan, 
follow-up  assessment  items  were  posed  to  the  brigade  to  re-assess  the  perceived  impact  that 
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Impact  data  collected  following  implementation  were  supplemented  with  data  collected  as 
the  unit  proceeded  through  its  yearly  training  plan.  Following  critical  events  in  their  plan, 
follow-up  assessment  items  were  posed  to  the  brigade  to  re-assess  the  perceived  impact  that 
BSTS  was  having  on  the  brigade.  The  vignettes  and  Janus  SIMEX  represented  the  first  two 
subsequent  assessment  opportunities. 

Following  the  conduct  of  two  vignettes,  brigade  and  battalion  staff  members  who  had  used 
their  BSTS  courses  were  asked  whether  or  not  the  BSTS  was  perceived  as  having  helped  them 
prepare  for  the  vignette-type  exercises  (Table  14).  Feedback  was  marginal,  but  no  respondents 
indicated  that  the  BSTS  had  not  been  helpful.  Of  participants  in  the  Mission  Analysis  vignette,  2 
of  5  agreed  that  the  BSTS  course  work  had  been  helpful;  3  were  neutral.  Of  participants  in  the 
Course  of  Action  Analysis  vignette,  the  only  participant  who  had  also  done  the  BSTS  agreed  that 
the  BSTS  had  helped  in  preparation  for  the  vignette. 


Table  14 

Perception  of  Utility  of  Battle  Staff  Training  System  (BSTS)  in  Preparing  for  Vignettes 


Questionnaire  Item 

Not  Helpful  or 
Slightly  Helpful 

Neutral 

Helpful  or 
Very  Helpful 

N 

How  much  BSTS  helped  in  preparing  for 

. . .  Mission  Analysis  Vignette 

0 

3  (60%) 

2  (40%) 

5 

. .  .Course  of  Action  Vignette 

0 

0 

1 

1 

Following  the  Janus  SIMEX,  participants  indicated  whether  or  not  the  BSTS  had  helped 
prepare  them  for  the  exercise.  Of  the  1 1  respondents,  8  were  neutral,  and  3  agreed  that  the  BSTS 
had  helped  them  prepare  for  the  Janus  SIMEX  (see  Table  15). 


Table  15 

Perception  of  Utility  of  Battle  Staff  Training  System  (BSTS)  in  Preparing  for  Janus-based 
Simulation  Exercise  (SIMEX) 


Questionnaire  Item 

Not  Helpful  or 
Slightly  Helpful 

Neutral 

Helpful  or 

Very  Helpful  N 

How  much  BSTS  helped  in  preparing  for 
Janus  SIMEX 

0 

8  (73%) 

3(27%)  11 

The  unit’s  trip  to  LTP  at  the  NTC  represented  the  next  opportunity  to  assess  the  impact  of 
BSTS.  The  LTP  participants  who  had  taken  part  in  BSTS  courses  were  presented  with  six 
questionnaire  items  on  the  subject.  Results  for  the  six  items  are  presented  in  Table  16. 

The  first  two  items  dealt  with  preparation  time,  and  inquired  to  what  extent  the  BSTS 
courses  helped  the  unit  use  their  preparation  time  more  effectively,  and  to  what  extent  the  BSTS 
courses  actually  saved  the  unit  time  in  preparing.  Results  on  these  two  items  were  generally 
positive,  with  11  of  14  on  both  items  recording  a  response  of  moderate,  great,  or  very  great 
extent. 
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Table  16 

Perceptions  of  the  Impact  of  Battle  Staff  Training  System  (BSTS)  on  Unit  Leader  Training 
Program  (LTP)  Performance 


Questionnaire  Item 

No  Extent  or 
Slight  Extent 

Moderate 

Extent 

Great  Extent  or 
Very  Great  Extent 

N 

Enabled  unit  to  use  preparation  time 
more  effectively 

3  (21%) 

9  (64%) 

2  (14%) 

14 

Saved  unit  preparation  time  for  the 

LTP 

3  (21%) 

4  (29%) 

7  (50%) 

14 

Provided  LTP-relevant  knowledge 
and  skills 

3  (21%) 

5  (36%) 

6  (43%) 

14 

Own  brigade  should  use 

4  (24%) 

11(65%) 

2(12%) 

17 

Other  brigades  should  use 

4  (25%) 

8  (50%) 

4  (25%) 

16 

Very  Harmful  or 
Slightly  Harmful 

Neutral 

Beneficial  or  Very 
Beneficial 

N 

How  BSTS  affected  LTP  performance 

1  (8%) 

4  (33%) 

7  (58%) 

12 

Another  item  asked  to  what  extent  the  BSTS  courses  had  provided  LTP-relevant  knowledge 
and  skills.  Participants  again  were  positive — 1 1  of  14  indicated  that  the  courses  had  provided 
knowledge  or  skills  to  a  moderate  or  great  extent. 

Participants  were  asked  again  whether  or  not  their  own  and  other  brigades  should  use  BSTS. 
A  full  77%  of  the  brigade  staff  respondents  agreed  their  own  unit  should  use  the  BSTS  courses 
for  future  training,  and  75%  stated  that  other  units  should  use  the  courses. 

A  summary  of  the  impact  of  BSTS  on  LTP  performance  was  collected  on  one  item  which 
asked  the  question:  “Overall,  how  did  BSTS  affect  your  performance  at  LTP?”  Of  the  12 
respondents,  one  indicated  that  the  BSTS  was  harmful,  and  another  four  respondents  said  that  the 
BSTS  was  neither  harmful  nor  beneficial.  Seven  respondents,  however,  said  that  the  courses  had 
been  beneficial  or  very  beneficial. 

The  remaining  two  training  events  were  the  BBSE  and  NTC  rotation.  After  the  BBSE, 
respondents  suggested  that  the  BSTS  had  a  modest  effect  in  helping  their  unit  prepare  for  the 
BBSE  (Table  17).  Of  20  respondents,  one  indicated  to  a  very  great  extent,  1 1  to  a  moderate 
extent,  5  to  a  slight  extent,  and  3  to  no  extent. 


Table  17 

Perceptions  of  the  Impact  of  Battle  Staff  Training  System  (BSTS)  on  Brigade  and  Battalion  Staff 
Exercise  Performance  (BBSE) 


Questionnaire  Item 

No  Extent  or 
Slight  Extent 

Moderate 

Extent 

Great  Extent  or 

Very  Great  Extent  N 

How  much  BSTS  helped  unit  prepare 
for  the  BBSE 

8  (40%) 

11  (55%) 

1  (5%)  20 
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The  concluding  event  of  the  unit’s  training  plan  was  the  NTC  rotation.  At  this  event, 
participants  were  asked  four  questions  regarding  the  impact  of  BSTS.  The  first  item  asked  to 
what  extent  the  BSTS  courses  had  contributed  to  improvements  made  in  the  brigade’s  staff 
process.  Three  out  of  5  brigade-level  respondents  indicated  to  a  moderate  extent,  and  4  of  5  task 
force  respondents  indicated  to  a  moderate  extent. 

The  second  post-NTC  question  asked  about  the  extent  to  which  respondents  would  use  the 
BSTS  courses  to  sustain  their  performance.  Feedback  on  this  item  was  mixed,  as  was  feedback 
on  the  third  item  that  asked  to  what  extent  other  brigades  or  task  forces  should  use  the  BSTS  for 
NTC  preparation  (see  Table  18). 


Table  18 

Perceptions  of  Battle  Staff  Training  System  Impact  Following  the  National  Training  Center 
(NTC)  Rotation 


Measure 

No  Extent  or 

Moderate 

Great  or  Very 

Respondent 

Slight  Extent 

Extent 

Great  Extent  N 

Improved  staff  process 

Brigade  staff 

2  (40%) 

3  (60%) 

0  5 

Task  force  staff 

1  (20%) 

4  (80%) 

0  5 

Own  brigade  or  task  force  should  use  to 
sustain  proficiency 

Brigade  staff 

1  (20%) 

1  (20%) 

3  (60%)  5 

Task  force  staff 

3  (50%) 

3  (50%) 

0  6 

Other  brigades  or  task  forces  should  use  for 
NTC  preparation 

Brigade  staff 

0 

1  (25%) 

3  (75%)  4 

Task  force  staff 

3  (50%) 

3  (50%) 

0  6 

Finally,  participants  were  asked  about  the  overall  impact  of  BSTS  on  NTC  performance. 

Both  brigade  and  task  force  staffs  were  positive  (Table  19)  with  2  of  4  (50%)  and  4  of  6  (67%), 
respectively,  indicating  that  the  BSTS  courses  had  been  beneficial,  while  2  of  4  (50%)  and  2  of  6 
(33%)  respondents  on  each  item  indicated  the  BSTS  had  been  neither  beneficial  nor  harmful. 

Table  19 

Perceptions  of  Battle  Staff  Training  System  Impact  on  National  Training  Center  Performance 

Harmful  or 

Beneficial  or 

Respondent 

Very  Harmful 

Neither 

Very  Beneficial  N 

Brigade  staff 

0 

2  (50%)  4 

Task  force  staff 

0 

2  (33%) 

4  (67%)  6 
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Battle  Staff  Training  System  Supportabilitv 


Recommendations  to  increase  the  supportability  centered  on  expanding  the  quantities  of  key 
course  materials,  making  tactical  graphics  readily  available,  the  training  management  system, 
and  program  startup.  The  problems  and  recommendations  are  presented  in  Table  20. 

Table  20 

Problems  and  Recommendations  Regarding  Battle  Staff  Training  System  Supportability 
Category 


Problem 

Basis  of  Issue 

Issuing  one  brigade  set  and  three  battalion  sets 
to  a  brigade  created  bottlenecks  when  staff 
assistants  were  tasked  to  complete  courses. 

Issuing  one  EMMii  CD-ROM  with  each 
brigade  or  battalion  set  created  bottlenecks 
when  several  students  in  a  unit  wanted  to  load 
software  on  unit-owned  or  student-owned 
computers  at  the  same  time. 

Issuing  one  brigade  and  one  battalion  COMPS 
CD-ROM  to  a  participating  brigade  created 
bottlenecks  when  several  students  were  ready 
to  take  the  COMPS  at  the  same  echelon. 

Accessibility  of  Overlays 

Providing  COMPS  overlays  only  in  Adobe 
Acrobat™  format  caused  delays  when  some 
students  had  to  locate  a  copy  of  Adobe  Acrobat 
Reader™. 

Printing  electronic  overlays  to  obtain 
hardcopies  led  to  “cut-and-paste”  assembly 
when  plotters  were  not  available. 

Training  Management  System 

Placing  EMMii  student  software  on  a  separate 
CD-ROM  increases  the  total  number  of  CD- 
ROMs  a  student  must  work  with. 

Awkward  and  confusing  procedures  for 
manual  transfer  of  performance  data  make  it 
easy  for  critical  errors  to  occur. 

Lack  of  capability  for  a  system  administrator  to 
recover  lost  or  corrupted  data  can  lead  to  an 
incomplete  performance  database. 


Recommendation 


Issuing  six  battalion  sets  to  the  brigade 
enabled  all  maneuver  and  support  battalions 
to  work  on  courses  at  the  same  time. 

Issuing  three  EMMii  CD-ROMs  with  each 
brigade  or  battalion  set  enabled  more  staff 
officers  to  load  system  software  without 
delay. 

Issuing  three  COMPS  CD-ROMs  with  each 
set  of  courseware  enabled  more  students  to 
take  the  end-of-course  comprehensive  exam 
at  the  same  time. 


Providing  a  copy  of  Adobe  Acrobat 
Reader™  shareware  on  each  student  CD- 
ROM  enabled  immediate  viewing  of 
overlays. 

Including  a  full-size  paper  copy  of  each 
COMPS  overlay  in  each  set  of  courseware 
made  it  easier  for  unit  personnel  to  reproduce 
overlays. 


Place  EMMii  student  files  on  the  CD-ROM 
set  containing  lesson  files. 

Modify  student  software  to  simplify  the 
transfer  process  and  add  safeguards  (e.g., 
confirmation  windows). 

Modify  the  software  suite  to  enable  a  system 
administrator  to  recover  data  from  a  student 
computer. 

table  continued 
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Table  20  (continued) 


Category 


Problem 


Recommendation 


In  a  local  area  network  environment,  the  TMS 
places  all  data  in  a  common  file,  precluding 
unit-specific  reports. 

Manual  transfer  of  data  in  stand-alone  mode 
places  an  extra  administrative  burden  on  unit 
personnel. 

Program  Start-up 

Shipping  courseware  sets  without  an 
inventory  of  components  that  make  up  a 
complete  set  can  confuse  unit  personnel. 

Lack  of  information  about  unit  computer 
assets  can  lead  to  implementation  difficulties. 

Failure  to  resolve  training  management  issues 
prior  to  use  of  BSTS  courses  can  degrade  the 
value  realized  from  the  training  program. 

The  TSP  materials  are  not  sufficiently 
detailed  to  enable  unit  personnel  to  conduct 
orientation  and  train-the-trainer  sessions. 


Migrate  the  software  suite  to  a  commercial  off 
the  shelf  (COTS)  product  offering  greater 
flexibility. 

Migrate  the  software  suite  to  a  COTS  product 
that  supports  uploading  of  data  via 
Intranet/Intemet. 

Include  an  inventory  list  showing  all 
components  of  a  set  (lesson  materials, 

COMPS  CD-ROMs,  operation  orders, 
overlays). 

Survey  computer  assets  of  target  units  in  time 
to  influence  fielding  plans  and  schedule. 

Inform  unit  leaders  of  training  management 
issues,  specifying  minimum  lead  time 
required  to  make  decisions. 

Expand  the  orientation  and  train-the-trainer 
components  of  the  TSP  to  include  a  narrative 
script. 

Create  support  base  of  fully  knowledgeable 
system  expert(s). 


BSTS/TMS  setup  and  technical  training 
require  system  knowledge  that  unit  personnel 
do  not  have. 


Note.  EMMii  =  Environment  for  MultiMedia  interactive  instruction;  COMPS  =  comprehensive  assessment 
component;  TMS  =  training  management  system;  COTS  =  commercial  off  the  shelf  software;  BSTS  =  Battle  Staff 
Training  System;  TSP  =  training  support  package. 


In  addition  to  the  foregoing,  observations  indicated  that  the  BSTS  should  include  an 
instructional  module  on  the  EMMii  CD-ROM  to  succinctly  describe  equipment  and  facilities 
requirements,  course  administration  duties,  training  audience,  map  requirements,  and  so  on. 
Additionally,  the  TMS  component  should  inform  unit  leaders  regarding  product  utilization  (e.g., 
guidelines  for  selecting  and  scheduling  courses  and  exercises).  A  fold-out  and  a  video  or 
interactive  multimedia  CD-ROM,  including  demonstration  clips  of  products  in  action,  would  be 
highly  desirable.  The  product  utilization  and  training  management  components  should  focus  on 
specific  actions  that  unit  leaders  need  to  take. 

Vignettes 

The  vignettes  were  implemented  by  Brigade  B  and  Brigade  C.  However,  Brigade  B,  which 
executed  four  vignettes,  utilized  very  few  of  the  TSP  materials.  Brigade  C  conducted  two 
vignettes,  generally  according  to  the  intended  model.  The  vast  majority  of  the  assessment  data 
comes  from  Brigade  C,  while  information  from  Brigade  B  is  used  to  reinforce  project 
conclusions  regarding  the  acceptability,  impact,  and  supportability  of  vignettes  in  general. 
Again,  7ATC  conducted  a  review  of  the  vignettes  during  the  timeframe  of  the  ISAT  project. 
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Points  made  during  the  review  are  included  in  the  assessment,  along  with  Brigade  C’s  vignette- 
related  feedback  obtained  during  its  Janus  SIMEX,  LTP,  BBSE,  and  NTC  rotation. 

Vignette  Acceptability 

Acceptability  data  were  gathered  following  vignette  implementations  and  during  the  7ATC 
review.  Participants  from  Brigade  C  provided  feedback  via  questionnaires  and  interviews  on  the 
currency  of  the  doctrine  taught  and  the  capacity  of  the  TSP  to  support  implementation. 

As  shown  in  Table  21,  participants  indicated  that  the  TSPs  were  consistent  with  current 
doctrine.  Most  inconsistencies  related  to  the  lack  of  a  Force  XXI  task  organization.  While  this 
should  be  corrected  as  updates  are  made  in  the  future,  none  of  the  noted  inconsistencies  proved 
to  be  significant  detractors  during  the  conduct  of  the  exercises.  The  trend  was  that  the  lack  of 
currency  was  noted  by  less  senior  officers,  and  dismissed  as  “below  the  noise  level”  by  more 
senior  officers. 

Table  21 


Participant  Perceptions  of  the  Doctrinal  Currency  of  Vignette  Materials 


Vignette 

Material 

Inconsistent  or 
Totally  Inconsistent 

Neither 

Consistent  or 
Totally  Consistent 

N 

Mission  Analysis  Vignette 

Operation  order/annexes 

1  (9%) 

1  (9%) 

9  (82%) 

11 

Overlays 

1  (11%) 

0 

8  (89%) 

9 

Job  aids 

0 

1  (14%) 

6  (86%) 

7 

Vignette  tasks 

0 

1  (20%) 

4  (80%) 

5 

Sample  products 

1  (17%) 

1  (17%) 

4  (66%) 

6 

Course  of  Action  Vignette 

Operation  order/annexes 

1  (25%) 

1  (25%) 

2  (50%) 

4 

Overlays 

1  (20%) 

1  (20%) 

3  (60%) 

5 

Job  aids 

1  (33%) 

1  (33%) 

1  (33%) 

3 

Vignette  tasks 

0 

1  (33%) 

2  (67%) 

3 

Sample  products 

0 

0 

4(100%) 

4 

Other  acceptance  criteria  included  how  easily  the  guides  were  used,  the  extent  to  which  the 
TSPs  were  viewed  as  supporting  successful  training,  and  whether  the  amount  of  time  required  by 
the  exercises  was  acceptable  (Table  22).  Most  of  the  respondents  found  it  easy  or  very  easy  to 
locate  information  in  the  Participant  Guide  (80%  for  one  vignette  and  50%  for  the  other). 
Respondents  also  indicated  that  the  TSPs  supported  implementation  and  that  the  time  needed  for 
vignette  preparation  was  generally  acceptable. 
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Table  22 


Perceptions  of  Usefulness  of  Vignette  Materials 


Questionnaire  Item 

How  easy  to  find  information 

Difficult  or 
Very  Difficult 

Neutral 

Easy  or  Very  Easy 

N 

...in  Mission  Analysis  Participant  Guide 

1  (10%) 

1  (10%) 

8  (80%) 

10 

...in  Course  of  Action  Participant  Guide 

0 

1  (50%) 

1  (50%) 

2 

Extent  to  which  materials  establish 
conditions  necessary  for  successful  training 

No  Extent  or 
Slight  Extent 

Moderate 

Extent 

Great  or  Very 
Great  Extent 

N 

...  in  Mission  Analysis  Vignette 

0 

1  (10%) 

9  (90%) 

10 

...in  Course  of  Action  Vignette 

1  (25%) 

2  (50%) 

1  (25%) 

4 

Acceptability  of  the  time  required  by 

Unacceptable 
or  Totally 
Unacceptable 

Neutral 

Acceptable  or 
Totally  Acceptable 

N 

. .  .Mission  Analysis  Vignette 

1  (8%) 

3  (25%) 

8  (67%) 

12 

. .  .Course  of  Action  Vignette 

0 

2  (50%) 

2  (50%) 

4 

The  unit  provided  a  number  of  comments  on  how  to  improve  the  vignette  TSPs.  They 
pointed  out  discrepancies  in  the  Tables  of  Organization  and  Equipment  (TOEs)  provided  in  the 
COA  Analysis  vignette  TSP.  This  was  due  to  changes  in  unit  TOEs  since  the  publication  of  the 
TSP.  Examples  noted  included  the  engineer  battalion  depicted  having  Combat  Engineer 
Vehicles,  which  are  no  longer  assigned.  And  a  field  artillery  battery  has  eight  guns,  as  opposed 
to  six  guns  under  the  new  Paladin  TOE. 

There  were  additional  shortfalls  noted  by  the  participants  in  the  tactical  materials  provided 
in  the  TSP.  Some  of  these  shortfalls  were:  the  use  of  the  old  double  block  to  depict  enemy  units 
on  overlays  as  opposed  to  the  new  diamond  depiction;  division  operations  appendix  included 
corps  sketch  depicting  mission,  but  not  one  for  division;  the  situation  update  did  not  give 
locations  of  units  outside  of  3rd  Brigade  area  of  operations;  and  the  Division  Support  Command 
(DISCOM)  plan  was  not  included  with  the  division  order. 

Additionally,  the  changes  shown  in  Table  23  were  recommended  for  the  COA  Analysis 
vignette. 
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Table  23 

Summary  of  Recommended  Course  of  Action  Vignette  Enhancements 
Area 

Recommendation 

Tactical  Materials 

•  Provide  ADA  graphics  in  1 :50,000  instead  of  1 :250,000 

•  Provide  complete  CSS  background  materials  and  information  (e.g.,  movement  rates, 
trafficability  of  routes) 

•  Fix  incomplete  CSS  map  graphics  (e.g.,  partial  main  supply  route) 

•  Revise  CSS  concept  to  ensure  consistency  with  OPORD  and  reflect  support  forward 

•  Add  newer  equipment  (such  as  Palletized  Loading  System)  to  CSS  materials 

•  Modify  artillery  Modified  TOE  and  scheme  of  maneuver  to  reflect  M109A6  Paladin 

•  Strengthen  and  focus  commander’s  guidance 

•  Expand  R&S  materials  and  information,  to  include  specific  commander’s  guidance 

•  Expand  enemy  information 

•  Focus  intelligence  products  on  NTC  OPFOR 
Assessment  Materials 

•  Add  criteria  for  comparing  courses  of  action  and  weighting  the  decision  matrix 

Note.  ADA  =  air  defense  artillery;  CSS  =  combat  service  support;  OPORD  =  operation  order;  TOE  =  Table  of 
Organization  and  Equipment;  R&S  =  reconnaissance  and  surveillance;  NTC  =  National  Training  Center;  OPFOR  = 
opposing  forces. 

Vignette  Impact 

Impact  data  were  concerned  with  measuring  the  effects  of  vignettes  on  individual  and  unit 
performance.  The  data  were  gathered  following  vignette  implementations  as  well  as  after  the 
Janus  SIMEX,  LTP,  BBSE,  and  NTC  rotation. 

After  the  implementation  of  two  vignettes,  participants  were  questioned  about  the  perceived 
effects  of  the  vignettes  on  the  individual  participants.  Vignettes  are  expected  to  allow  for  the 
practice  of  warfighting  skills  and  improve  task  understanding,  task  performance,  and  teamwork 
skills.  Thus,  questions  were  framed  around  these  expectations.  For  both  vignettes,  responses 
were  positive  (see  Table  24).  Respondents  believed  the  vignettes  provided  practice 
opportunities,  and  that  they  were  able  to  improve  (generally  to  a  moderate  or  great  extent)  both 
understanding  and  performance  as  intended. 
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Table  24 

Perceptions  of  the  Impact  of  Vignettes  on  Individual  Users 


Vignette 


Area  of  Impact 


No  Extent  or 
Slight  Extent 


Moderate 

Extent 


Great  Extent  or 
Very  Great 
Extent 


N 


Mission  Analysis  Vignette 

Enabled  practice  on  warfighting  techniques 
Improved  understanding  of  tasks 
Improved  teamwork  skills 
Improved  task  performance 
Course  of  Action  Vignette 

Enabled  practice  of  warfighting  techniques 
Increased  task  understanding 
Improved  teamwork  skills 
Improved  task  performance 


(08%) 

4  (33%) 

7  (58%) 

12 

(17%) 

7  (58%) 

3  (25%) 

12 

(17%) 

5  (42%) 

5  (42%) 

12 

(17%) 

8  (67%) 

2(17%) 

12 

0 

2  (40%) 

3  (60%) 

5 

0 

4  (80%) 

1  (20%) 

5 

0 

1  (20%) 

4  (80%) 

5 

0 

2  (40%) 

3  (60%) 

5 

Questionnaire  items  also  addressed  the  effects  of  the  vignettes  on  the  unit  and  its  training. 
Responses  to  these  items  are  contained  in  Table  25.  Most  of  the  training  audience  respondents 
felt  that  the  Mission  Analysis  vignette  helped  the  brigade  (a)  meet  its  staff  training  goals  to  a 
moderate  extent  or  greater  and  (b)  become  better  prepared  for  the  LTP.  Similarly,  a  high 
percentage  (75%  or  greater)  indicated  that  the  vignette  enhanced  the  brigade’s  combat  readiness 
to  a  moderate  extent  or  greater.  Perceptions  of  the  Course  of  Action  vignette  were  less 
favorable,  but  still  generally  positive  regarding  meeting  training  goals  and  preparing  for  LTP. 

Table  25 


Perceptions  of  the  Impact  of  Vignettes  on  the  Unit 


Vignette 

Area  of  Impact 

No  Extent  or 
Slight  Extent 

Moderate 

Extent 

Great  Extent  or 
Very  Great  Extent 

N 

Mission  Analysis  Vignette 

Help  meet  staff  training  goals 

0 

3  (25%) 

9  (75%) 

12 

Better  prepare  unit  for  Leader 

0 

3  (25%) 

9  (75%) 

12 

Training  Program 

Enhance  unit  combat  readiness 

0 

3  (25%) 

9  (75%) 

12 

Save  unit  training  time 

0 

4  (33%) 

8  (67%) 

12 

Save  unit  training  funds 

1  (8%) 

6  (50%) 

5  (42%) 

12 

Course  of  Action  Vignette 

Help  meet  staff  training  goals 

0 

3  (75%) 

1  (25%) 

4 

Better  prepare  unit  for  Leader 

0 

3  (75%) 

1  (25%) 

4 

Training  Program 

Enhance  unit  combat  readiness 

1  (25%) 

2  (50%) 

1  (25%) 

4 

Save  unit  training  tine 

1  (25%) 

1  (25%) 

2  (50%) 

4 

Save  unit  training  fund 

2  (50%) 

0 

2  (50%) 

4 
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Regarding  the  future  utility  of  the  vignettes,  participants  were  asked  to  rate  the  extent  to 
which  the  benefits  outweighed  the  costs  and  whether  their  brigade  and  other  brigades  should  use 
the  vignettes  in  the  future.  Results  were  positive,  as  provided  in  Table  26,  with  most  respondents 
agreeing  that  both  vignettes  should  be  used  in  the  future.  These  results  indicate  strongly  that  the 
training  audience  perceived  the  training  value,  both  past  and  future,  in  the  vignette  exercises. 


Table  26 

Perceptions  of  the  Future  Utility  of  the  Vignettes 


Vignette 

Measure 

No  Extent  or 
Slight  Extent 

Moderate 

Extent 

Great  or  Very 
Great  Extent 

N 

Mission  Analysis  Vignette 

Benefits  outweigh  costs 

0 

3  (25%) 

9  (75%) 

12 

Own  brigade  should  use 

0 

0 

12(100%) 

12 

Other  brigades  should  use 

0 

0 

12  (100%) 

12 

Course  of  Action  Vignette 

Benefits  outweigh  costs 

0 

2  (50%) 

2  (50%) 

4 

Own  brigades  should  use 

2  (40%) 

1  (20%) 

2  (40%) 

5 

Other  brigades  should  use 

1  (20%) 

2  (40%) 

2  (40%) 

5 

The  next  opportunity  to  assess  the  effects  of  the  vignettes  was  during  the  Janus  SIMEX 
conducted  by  Brigade  C.  Participants  were  asked  whether  they  agreed  or  disagreed  that  the 
vignette  had  enabled  them  to  get  more  out  of  the  Janus  SIMEX.  Of  14  respondents,  6  agreed  that 
the  vignettes  had  helped  (see  Table  27).  Eight  neither  agreed  nor  disagreed. 


Table  27 

Perception  of  Utility  of  Vignettes  in  Preparing  for  Janus-based  Exercise 

Questionnaire  Item 

Not  Helpful  or 
Slightly  Helpful 

Neutral 

Helpful  or 

Very  Helpful  N 

How  much  vignettes  helped  in  preparing  for 
Janus  SIMEX 

0 

8  (57%) 

6  (43%)  14 

Note.  SIMEX  =  simulation  exercise. 


Following  the  brigade’s  LTP,  several  questions  were  asked  about  the  impact  of  vignettes  on 
LTP  train-up  and  performance.  As  shown  in  Table  28, 87%  of  respondents  said  that  the 
vignettes  had  provided  LTP-relevant  knowledge  and  skills  to  a  moderate  or  great  extent.  Eighty 
percent  said  that  the  vignettes  allowed  them  to  use  their  preparation  time  more  effectively,  and 
67%  said  that  the  vignettes  had  helped  the  unit  save  preparation  time.  When  asked  to  what 
extent  their  own  and  other  brigades  should  use  the  vignettes  to  prepare  for  future  LTPs,  81%  of 
the  brigade  staff  respondents  agreed  their  own  unit  should  use  the  vignettes  for  future  training, 
and  69%  felt  other  units  should  use  the  vignettes. 
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Table  28 

Perceptions  of  Vignette  Impact  on  Unit  Leader  Training  Program  (LTP)  Performance 


Area  of  Impact 

No  Extent  or 
Slight  Extent 

Moderate 

Extent 

Great  Extent  or 
Very  Great  Extent 

N 

Enabled  unit  to  use  preparation  time  more 
effectively 

3  (20%) 

4  (27%) 

8  (53%) 

15 

Saved  unit  preparation  time  for  the  LTP 

5  (33%) 

5  (33%) 

5  (33%) 

15 

Provided  LTP-relevant  knowledge  and 
skills 

2(13%) 

6  (40%) 

7  (47%) 

15 

Own  brigade  should  use 

3  (19%) 

5  (31%) 

8  (50%) 

16 

Other  brigades  should  use 

5(31%) 

4  (25%) 

7  (44%) 

16 

Following  the  unit’s  BBSE  implementation,  participants  were  asked  to  what  extent  the 
vignettes  helped  prepare  them  for  the  BBSE  (see  Table  29).  Of  8  respondents,  5  said  to  a 
moderate  extent  and  3  to  a  great  or  very  great  extent.  No  respondents  selected  the  slight  extent 
or  not  at  all  options. 

Table  29 

Perceptions  of  the  Impact  of  Vignettes  on  Brigade  and  Battalion  Staff  Exercise  (BBSE) 
Performance 


No  Extent  or  Moderate  Great  Extent  or 


Questionnaire  Item 

Slight  Extent 

Extent 

Very  Great  Extent 

N 

How  much  vignettes  helped  unit  prepare 
for  the  BBSE 

0 

5  (62%) 

3  (38%) 

8 

The  unit’s  NTC  rotation  provided  the  final  assessment  opportunity.  Participants  were  asked 
about  their  own  and  other  units’  future  use  of  the  vignettes  in  preparation  for  NTC  rotations,  as 
well  as  about  the  overall  impact  of  the  vignettes  on  NTC  performance.  As  shown  in  Table  30, 
ratings  for  future  use  of  the  vignettes  were  high,  while  ratings  of  the  impact  of  vignettes  on  NTC 
performance  were  mixed. 

Table  30 


Perceptions  of  Future  Utility  of  Vignettes  for  Preparing  for  National  Training  Center  Rotations 


Measure 

No  Extent  or 

Moderate 

Great  Extent  or 

Respondent 

Slight  Extent 

Extent 

Very  Great  Extent 

N 

Own  brigade  should  use  to  sustain 
proficiency 

Brigade  staff 

0 

2  (40%) 

3  (60%) 

5 

Task  force  staff 

0 

1  (50%) 

1  (50%) 

2 

Other  brigades  should  use  for  NTC 
preparation 

Brigade  staff 

0 

2  (40%) 

3  (60%) 

5 

Task  force  staff 

0 

3  (75%) 

1  (25%) 

4 
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Table  31  shows  that,  among  combined  brigade  and  task  force  participants,  4  of  9 
respondents  (45%)  said  the  vignettes  were  neither  harmful  nor  beneficial,  while  the  5 
respondents  (55%)  said  the  vignettes  were  beneficial  to  their  NTC  performance. 


Table  31 

Perceptions  of  Vignette  Impact  on  National  Training  Center  Performance 


Respondent 

Harmful  or 
Very  Harmful 

Neither 

Beneficial  or 
Very  Beneficial 

N 

Brigade  staff 

0 

4  (80%) 

5 

Task  force  staff 

0 

3  (75%) 

1  (25%) 

4 

Vignette  Supportabilitv 

This  section  contains  recommendations  regarding  the  supportability  and  use  of  the  vignettes. 
The  information  is  a  consolidation  of  unit  feedback  and  ISAT  team  observations  and  conclusions 
and  unit  feedback.  Many  of  the  comments  and  observations  focused  on  the  train-up  for  the  use 
of  vignettes  (orientation  briefing  and  workshop),  which  took  place  only  days  before  the 
scheduled  execution  of  the  first  vignette.  Other  attention  was  centered  on  the  tactical  products, 
the  job  aids,  and  the  terrain  database.  The  most  frequent  comments  are  summarized  in  Table  32. 

Table  32 

Problems  and  Recommendations  Regarding  Vignette  Supportability 


Category 

Problem 


Recommendation 


Vignette  Start-Up  Training 
Vignettes  not  integrated 
into  unit’s  training  plan 


Unit  leaders  did  not 
understand  how  to 
match  vignettes  to 
training  need 


Conduct  at  least  a  month  prior  to  vignette  implementation.  This  would 
permit  brigade  leaders  to  assess  their  current  training  needs,  understand 
the  content  of  the  vignettes,  and  select/schedule  high-payoff  vignettes. 

Restructure  the  vignette  orientation  briefing  and  workshop  training  to 
enable  the  brigade  to  assess  their  current  level  of  expertise,  comprehend 
the  content  of  the  vignettes,  and  determine  the  which  vignettes  should 
be  used  for  their  training,  and  when  the  vignettes  should  be  scheduled. 

Modify  briefing  to  emphasize  purpose  of  vignettes,  that  they  are 
designed  to  allow  use  of  staff  procedures. 

Organize  vignettes  and  synchronize  scenarios  to  represent  discrete  steps 
in  the  plan-prepare-execute  sequence  of  an  integrated  scenario  story 
line.  The  family  of  resulting  exercises  would  be  sequentially  related, 
one  event  leading  to  or  setting  the  stage  for  the  next.  The  major  tasks 
or  challenges  facing  a  staff  would  be  represented  in  an  orderly  array, 
and  specific  exercises  could  be  selected  based  on  known  weaknesses. 

table  continues 
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Table  32  (continued) 


Category 

Problem 


Recommendation 


Tactical  Materials 

Tactical  materials  do  not 
reflect  actual  equipment 
and  task  organization 

Job  Aids 

Purpose  of  job  aids  not 
clear,  confusing  when 
compared  to  BSTS  job 
aids 


Terrain  Database 


Include  electronic  copies  of  the  orders  on  CD-ROM,  so  unit  staff  could 
modify  the  files  and  then  print  the  unit-specific  products. 


Discuss  differences  with  BSTS  job  aids,  which  are  primarily 
information  documents,  cheat  sheets  for  combat  ratios,  movement 
ratios,  computing  engineer  work  timetables. 

Add  doctrinal  charts  showing  the  military  decision-making  process  in 
Field  Manual  101-5  (Department  of  the  Army,  1997).  Vignette  job  aids 
present  formats  and  techniques  used  during  planning,  many  based  on 
those  developed  by  units  or  Combat  Training  Centers. 

Update  and  synchronize  the  vignette,  BSE/BBSE  and  BSTS  job  aids  to 
reflect  current  doctrine,  terms  and  graphics  (enemy  capabilities,  new 
equipment  such  as  Paladin  and  Palletized  Loading  System,  new  combat 
service  support  concepts,  latest  Modified  TOE). 


BBS  terrain  does  not  Update  BBS  terrain  database  files  (and  overlays)  to  reflect  WGS84, 

conform  to  current  map  100,000  meter  grid  zone  designators  so  we  don’t  have  to  reconfigure 
standards  map  boards  to  accommodate  nonstandard  overlays  or  chase  hard-to- 

find,  outdated  map  sheets.  In  the  interim,  add  errata  sheet  to  the 
training  support  packages  detailing  conversion  of  100,000  meter  grid 
zone  designators  from  WGS84  system  to  Defense  Mapping  Agency, 
Clarke  1866  spheroid,  system. 

Note.  BSTS  =  Battle  Staff  Training  System;  BSE  =  Brigade  Staff  Exercise;  BBSE  =  Brigade  and  Battalion  Staff 
Exercise;  TOE=  Table  of  Organization  and  Equipment. 


All  of  these  recommendations  are  worthy  of  further  examination,  but  have  not  been 
evaluated  within  this  study.  For  example,  the  recommendation  to  include  electronic  copies  of  the 
orders  on  CD-ROM,  so  unit  staff  could  modify  the  files  and  then  print  the  unit-specific  products, 
would  hardly  be  risk-free.  The  tactical  materials  (OPORDS,  overlays,  message  lists,  decision 
support  templates,  synchronization  matrixes,  etc.)  are  all  interdependent,  and  ensuring  that  they 
maintain  proper  alignment  would  demand  significant  effort  and  attention  to  detail.  A  “wizard” 
that  draws  templates  from  a  TSP  database  and  leads  the  unit  trainer  through  required  steps  would 
appear  sensible,  but  it  would  not  guarantee  completeness  of  modifications.  Who  would  perform 
quality  control  checks  of  modified  materials?  Who  would  accept  responsibility  if  flaws  in 
modified  products  led  to  degraded  training  exercises?  This  area  and  possible  solutions  like  the 
others  in  Table  32,  merit  further  analysis  and  research  to  serve  Army  needs. 

Janus  Simulation  Exercise 


The  Janus  SIMEX  (described  in  Section  2  of  this  report),  while  not  a  product  of  the 
FXXITP,  was  included  in  Brigade  C’s  training  schedule  in  place  of  the  BSE.  The  BSE  that  had 


49 


been  scheduled  became  infeasible  due  to  the  lack  of  access  to  the  BBS.  The  assessment  of  the 
Janus  SIMEX  provides  for  a  loose  comparison  between  the  acceptability,  impact,  and 
supportability  of  an  exercise  developed  upon  unfolding  training  requirements  and  the  other 
products  of  the  FXXITP  that  were  developed  for  more  general  usage.  Data  on  this  product  may 
also  shed  light  on  the  potential  for  providing  rapid-response  training  development  support. 

Janus  Simulation  Exercise  Acceptability 

Doctrinal  currency  was  assessed  by  the  primary  and  secondary  training  audiences  (see  Table 
33).  Only  a  third  to  a  half  of  the  primary  training  audience  found  the  tactical  materials 
(OPORDs,  overlays,  warning  orders  (W ARNOs)  and  fragmentary  orders  (FRAGOs)  consistent 
or  totally  consistent  with  current  doctrine,  while  the  corresponding  figure  for  the  Performance 
Objectives  was  67%.  On  the  other  hand,  70%  or  more  of  the  secondary  training  audience  (battle 
captains,  assistant  staff  officers,  section  NCOs)  found  all  those  TSP  components  consistent  or 
totally  consistent  with  current  doctrine.  Overall,  these  findings  suggest  (not  surprisingly)  that  it 
is  difficult  to  ensure  doctrinal  currency  when  reaction  time  is  short. 


Table  33 

Perceptions  of  Doctrinal  Currency  of  Janus  Simulation  Exercise  Materials 


Respondent  Group 

Material 

Inconsistent  or 
Totally  Inconsistent 

Neither 

Consistent  or 
Totally  Consistent 

N 

Primary  Training  Audience 

OPORD/Annexes 

8  (62%) 

1  (8%) 

4(31% 

13 

Overlays 

3  (25%) 

3  (25%) 

6  (50%) 

12 

WARNOs/FRAGOs 

5  (38%) 

3  (23%) 

5  (38%) 

13 

Performance  Objectives 

1  (11%) 

2  (22%) 

6  (67%) 

9 

Secondary  Training  Audience 

OPORD/Annexes 

2(18%) 

1  (09%) 

8  (73%) 

11 

Overlays 

2(18%) 

1  (09%) 

8  (73%) 

11 

WARNOs/FRAGOs 

0 

2(18%) 

9  (82%) 

11 

Performance  Objectives 

1  (12%) 

1  (12%) 

6  (75%) 

8 

Note.  OPORD  =  operations  order;  WARNO  =  warning  order;  FRAGO  =  fragmentary  order. 


About  half  of  the  training  audience  participants  (primary  and  secondary)  reported  they  used 
the  participant  guide  and  the  POs,  while  all  participants  used  the  division/brigade  OPORD. 

Table  34  contains  the  responses  of  primary  and  secondary  training  audience  members  regarding 
how  easily  they  were  able  to  find  information  in  their  participant  guides.  Among  the  primary 
training  audience,  about  half  indicated  it  was  easy  to  find  information  in  the  participant  guide. 
About  a  third  of  the  secondary  training  audience  reported  it  was  easy  to  find  information  in  the 
guide.  Participants  were  also  asked  about  the  extent  to  which  the  information  and  materials 
provided  for  the  exercise  established  the  conditions  for  successful  training.  Responses  to  this 
item  were  split  among  slight,  moderate,  and  great  extent.  When  asked  about  the  acceptability  of 
the  amount  of  time  spent  on  preparation  for  the  Janus  SIMEX,  respondents  in  the  primary 
training  audience  were  generally  positive,  indicating  the  time  was  acceptable.  The  secondary 
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audience  reported  similarly.  As  shown  later  in  this  section,  these  ratings  were  similar  to  those 
made  regarding  the  acceptability  of  the  preparation  time  spent  for  the  BBSE. 


Table  34 

Perceptions  of  Usefulness  of  Janus  Simulation  Exercise  Materials 


Questionnaire  Item 

How  easy  to  find  information  in 
guides  by 

Difficult  or  Very 
Difficult 

Neither 

Easy  or  Did  not 

Very  Easy  use 

N 

. .  .Primary  Training  Audience 

0 

4  (27%)  8  (53%) 

15 

. .  .Secondary  Training  Audience 

1  (8%) 

3  (23%) 

2(15%)  7(54%) 

13 

Extent  to  which  materials  establish 
conditions  necessary  for  successful 
training 

No  Extent  or  Slight 
Extent 

Moderate 

Extent 

Great  Extent  or  Very 
Great  Extent 

N 

Primary  training  audience 

5  (33%) 

5  (33%) 

5  (33%) 

15 

Secondary  training  audience 

3  (25%) 

5  (42%) 

4  (33%) 

12 

Acceptability  of  the  time  required 
by  the  training 

Totally 

Unacceptable  or 
Unacceptable 

Neither 

Totally  Acceptable  or 
Acceptable 

N 

Primary  training  audience 

2(18%) 

3  (27%) 

6  (54%) 

11 

Secondary  training  audience 

4(13%) 

3  (20%) 

8  (53%) 

15 

Janus  Simulation  Exercise  Impact 

The  perceived  impact  of  the  Janus  SIMEX  on  the  individual  participants  was  not  out  of  line 
with  the  perceived  impact  of  other  FXXITP  products.  Table  35  shows  that  more  than  half  of  the 
primary  training  audience  respondents  felt  (to  a  great  or  very  great  extent)  that  the  exercise 
enabled  them  to  practice  techniques  they  would  use  in  battle.  The  corresponding  figures  for  the 
secondary  training  audience  dropped  slightly  to  39%.  Among  the  primary  training  audience, 
60%  or  more  reported  improvements  (to  a  moderate  extent  or  greater)  in  their  task 
understanding,  task  performance,  and  teamwork  skills  job  skills/abilities,  and  self-confidence. 
Among  the  secondary  training  audience,  70%  or  more  reported  the  same  improvements. 
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Table  35 

Perceptions  of  the  Impact  of  the  Janus  Simulation  Exercise  on  Individual  Users 


Respondent 

Area  of  Impact 

No  Extent  or 
Slight  Extent 

Moderate 

Extent 

Great  Extent  or 
Very  Great  Extent 

N 

Primary  Training  Audience 

Enabled  practice  on  battle  techniques 

3  (20%) 

4  (275) 

8  (53%) 

15 

Improved  understanding  of  own  tasks 

5  (33%) 

4  (27%) 

6  (40%) 

15 

Improved  own  teamwork  skills 

1  (7%) 

6  (40%) 

8  (53%) 

15 

Improved  ability  to  perform  own  tasks 

6  (40%) 

3  (20%) 

6  (40%) 

15 

Secondary  Training  Audience 

Enabled  practice  on  battle  techniques 

1  (8%) 

7  (54%) 

5  (38%) 

13 

Improved  understanding  of  own  tasks 

2  (25%) 

6  (46%) 

5  (38%) 

13 

Improved  own  teamwork  skills 

1  (9%) 

5  (45%) 

5  (45%) 

11 

Improved  ability  to  perform  own  tasks 

3  (23%) 

5  (38%) 

5  (38%) 

13 

The  great  majority  of  primary  and  secondary  training  audience  respondents  felt  that  the 
Janus  SIMEX  contributed  materially  to  the  unit  training  program.  Table  36  shows  that  80%  or 
more  of  the  primary  training  audience  and  100%  of  the  secondary  training  audience  reported  that 
the  exercise  helped  the  brigade  to  (a)  meet  its  staff  training  goals  to  a  moderate  extent  or  greater 
and  (b)  become  better  prepared  for  the  LTP.  Among  both  training  audiences,  80%  or  more 
perceived  that  the  brigade’s  combat  readiness  benefited  from  the  Janus  SIMEX  to  a  moderate 
extent  or  greater.  More  than  75%  of  all  respondents  estimated  that  the  exercise  saved  their  unit 
substantial  training  time  and  funds. 


Table  36 

Perceptions  of  Impact  of  the  Janus  Simulation  Exercise  on  the  Unit 


Respondent 

Area  of  Impact 

No  Extent  or 
Slight  Extent 

Moderate 

Extent 

Great  Extent  or 
Very  Great  Extent 

N 

Brigade  Training  Audience 

Help  meet  staff  training  goals 

1  (7%) 

4  (27%) 

10  (67%) 

15 

Better  prepare  unit  for  Leader  Training 

3(20) 

3  (20%) 

9  (60%) 

15 

Program 

Enhance  unit  combat  readiness 

3(20) 

2  (13%) 

10  (67%) 

15 

Save  unit  training  time 

2  (13%) 

4  (26%) 

9  (60%) 

15 

Save  unit  training  funds 

1  (7%) 

7  (50%) 

6  (43%) 

14 

Task  Force  Training  Audience 

Help  meet  staff  training  goals 

0 

5  (38%) 

8  (62%) 

13 

Better  prepare  unit  for  Leader  Training 

0 

4(31%) 

9  (69%) 

13 

Program 

Enhance  unit  combat  readiness 

1  (8%) 

6  (50%) 

5  (42%) 

12 

Save  unit  training  time 

3  (25%) 

3  (25%) 

6  (50%) 

12 

Save  unit  training  funds 

2  (17%) 

1  (8%) 

9(75%) 

12 
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As  Table  37  shows,  most  of  the  training  audience  respondents  stated  (to  a  moderate  extent 
or  greater)  that  their  own  unit  should  use  the  Janus  SIMEX  for  future  training  and  that  other  units 
should  use  the  exercise  as  well.  More  than  75%  estimated  that  the  benefits  outweighed  the  cost 
of  the  exercise. 


Table  37 

Perceptions  of  Future  Utility  of  the  Janus  Simulation  Exercise 


Respondent 

Who  Should  Use 

No  Extent  or 
Slight  Extent 

Moderate 

Extent 

Great  Extent  or 
Very  Great  Extent 

N 

Brigade  Training  Audience 

Benefits  outweigh  costs 

2(13%) 

5  (33%) 

8  (53%) 

15 

Own  brigade  should  use 

2(13%) 

6  (40%) 

7  (47%) 

15 

Other  brigades  should  use 

2(13%) 

6  (40%) 

7  (47%) 

15 

Task  Force  Training  Audience 

Benefits  outweigh  costs 

2(18%) 

3  (27%) 

6  (55%) 

11 

Own  task  force  should  use 

2(17%) 

8  (67%) 

2  (17%) 

12 

Other  task  force  should  use 

2(17%) 

8  (67%) 

2(17%) 

12 

The  next  opportunity  to  assess  the  effects  of  the  Janus  SIMEX  was  at  the  end  of  the  LTP. 

At  that  time,  Janus  SIMEX  participants  were  asked  questions  regarding  the  impact  of  the  SIMEX 
on  their  LTP  performance.  When  asked  about  the  effects  on  preparation  time  and  effectiveness 
(e.g.,  provided  LTP-relevant  knowledge  and  skills),  respondents  provided  mixed  feedback.  On 
all  three  items,  however,  more  than  60%  of  respondents  indicated  the  Janus  SIMEX  contributed 
(to  a  moderate  or  great  extent)  to  preparation  (see  Table  38).  Additionally,  the  LTP  participants 
who  had  participated  in  the  Janus  SIMEX  were  asked  about  the  extent  to  which  their  and  other 
units  should  use  the  Janus  SIMEX  in  preparation  for  the  LTP.  Of  the  17  respondents,  88%  of  the 
unified  training  audience  agreed  their  own  unit  should  use  the  Janus  SIMEX  for  future  training 
in  preparation  for  LTP,  and  the  same  proportion  felt  other  units  should  use  it  as  well. 

The  LTP  participants  were  also  asked  for  their  perceptions  of  the  overall  impact  of  the  Janus 
SIMEX  on  their  LTP  performance.  The  response  was  overwhelmingly  positive  (see  Table  38), 
as  15  of  16  indicated  the  Janus  SIMEX  had  been  beneficial  or  very  beneficial  and  one  respondent 
indicated  the  SIMEX  was  neither  harmful  nor  beneficial. 
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Table  38 

Perceptions  of  Impact  of  the  Janus  Simulation  Exercise  on  Unit  Leader  Training  Program 
Performance 


Area  of  Impact 

No  Extent  or 
Slight  Extent 

Moderate 

Extent 

Great  Extent  or 
Very  Great  Extent 

N 

Enabled  unit  to  use  preparation  time 
more  effectively 

4  (27%) 

5  (33%) 

6  (40%) 

15 

Saved  unit  preparation  time  for  the 

LTP 

4  (29%) 

3  (21%) 

7  (50%) 

14 

Provided  LTP-relevant  knowledge 
and  skills 

5  (33%) 

1  (7%) 

9  (60%) 

15 

Own  brigade  should  use 

2(12%) 

5  (29%) 

10(59%) 

17 

Other  brigades  should  use 

2(12%) 

6  (35%) 

9  (53%) 

17 

Very  Harmful  or 
Slightly  Harmful 

Neutral 

Beneficial  or  Very 
Beneficial 

N 

How  Janus  SIMEX  affected  LTP 
performance 

0 

1  (6%) 

15  (94%) 

16 

Note.  LTP  =  leader  training  program;  SIMEX 

=  simulation  exercise. 

Before  their  NTC  rotation,  the  brigade  participated  in  an  implementation  of  the  BBSE. 
Following  that  implementation,  brigade  and  task  force  staff  estimated  the  extent  to  which  the 
Janus  SIMEX  had  helped  them  prepare  for  the  BBSE.  Results,  presented  in  Table  39,  were 
positive  as  85%  indicated  that  the  Janus  SIMEX  had  been  helpful  (to  a  moderate  or  great  extent) 
in  BBSE  preparation. 

Table  39 

Perceptions  of  the  Impact  of  the  Janus  Simulation  Exercise  on  Brigade  and  Battalion  Staff 
Exercise  Performance 

Respondent 

No  Extent  or 
Slight  Extent 

Moderate 

Extent 

Great  Extent  or 
Very  Great  Extent 

N 

Brigade  staff 

1  (13%) 

2  (25%) 

5  (62%) 

8 

Task  force  staff 

2(15%) 

7  (54%) 

4(31%) 

13 

After  the  brigade’s  NTC  rotation,  Janus  SIMEX  participants  were  asked  for  their 
perceptions  of  the  overall  impact  of  the  Janus  SIMEX  on  their  NTC  performance.  As  shown  in 
Table  40,  despite  the  low  response  rates,  the  ratings  were  generally  positive  from  the  brigade 
level  training  audience,  and  ambivalent  from  the  task  force  respondents. 
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Table  40 

Perceptions  of  the  Impact  of  the  Janus  Simulation  Exercise  on  National  Training  Center 
Performance 


Respondent 

Harmful  or 
Very  Harmful 

Neither 

Beneficial  or 
Very  Beneficial 

N 

Brigade  staff 

1  (25%) 

0 

3  (75%) 

4 

Task  force  staff 

0 

3  (75%) 

1  (25%) 

4 

Janus  Simulation  Exercise  Supportabilitv 

The  end-of-SIMEX  questionnaires  asked  the  training  audience  whether  the  IS  AT  team 
support  (tactical  and  Janus  experts  performing  many  of  the  exercise  functions)  was  necessary  or 
not.  As  seen  in  Table  41,  all  of  the  primary  training  audience  respondents  and  about  two-thirds 
of  the  secondary  training  audience  judged  that  the  external  support  was  necessary.  This 
underscores  the  importance  of  on-site  personnel  who  understand  the  TSP  intimately  when  quick- 
reaction  exercises  are  crafted  to  meet  unexpected  unit  training  needs. 


Table  41 

Perceptions  of  the  Necessity  of  the  Support  Provided  for  the  Janus  Simulation  Exercise 


Respondent 

Measure 

No  Extent  or 
Slight  Extent 

Moderate 

Extent 

Great  Extent  or 
Very  Great  Extent 

N 

Primary  Training  Audience 

Extent  to  which  ISAT  team  was  needed 

0 

0 

13(100%) 

13 

Extent  to  which  Janus  staff  was  needed 

0 

0 

13  (100%) 

13 

Secondary  Training  Audience 

Extent  to  which  ISAT  team  was  needed 

0 

3  (38%) 

5  (62%) 

8 

Extent  to  which  Janus  staff  was  needed 

0 

2  (33%) 

4  (67%) 

6 

Brigade  and  Battalion  Staff  Exercise 

The  BBSE  was  conducted  only  once  under  the  auspices  of  the  project,  by  Brigade  C,  mid¬ 
way  between  their  LTP  visit  and  their  NTC  rotation.  The  data  presented  here  were  gathered 
during  and  after  the  BBSE  implementation,  and  after  the  NTC  rotation  itself. 

Brigade  and  Battalion  Staff  Exercise  Acceptability 

Table  42  summarizes  the  perceptions  of  the  training  audience  and  support  personnel 
regarding  the  doctrinal  consistency  of  various  BBSE  training  materials.  The  brigade  staff 
training  audience  respondents  gave  the  key  materials  consistently  high  marks  (89%  to  100% 
reporting  that  the  materials  were  doctrinally  correct);  while  generally  positive,  the  task  force 
participants’  ratings  were  somewhat  lower  (65%  to  80%  reported  that  the  materials  were 
doctrinally  correct). 


55 


Table  42 

Perceptions  of  the  Doctrinal  Currency  of  Brigade  and  Battalion  Staff  Exercise  Materials 


Respondent 

Material 

Inconsistent  or 
Totally  Inconsistent 

Neither 

Consistent  or 
Totally  Consistent 

N 

Brigade  Training  Audience 

Operation  order/annexes 

0 

0 

9  (100%) 

9 

Overlays 

0 

0 

9  (100%) 

9 

Initial  situation  package 

0 

1  (11%) 

8  (89%) 

9 

WARNOs/FRAGOs 

0 

0 

9  (100%) 

9 

Performance  objectives 

0 

0 

6  (100%) 

6 

Task  Force  Training  Audience 

Operation  order/ Annexes 

2(13%) 

1  (7%) 

12  (80%) 

15 

Overlays 

1  (7%) 

2  (13%) 

12  (80%) 

15 

Initial  situation  package 

2(14%) 

3  (21%) 

9  (65%) 

14 

WARNOs/FRAGOs 

1  (6%) 

3  (19%) 

12(75%) 

16 

Performance  objectives 

1  (7%) 

2(14%) 

11  (79%) 

14 

Note.  WARNO  =  warning  order;  FRAGO  =  fragmentary  order. 


Likewise,  observers’  ratings  of  the  PO  materials  were  consistently  high.  Between  67%  and 
87%  of  the  BLUFOR  roleplayers  and  interactors  as  well  as  EXCON  personnel  rated  the  tactical 
materials  and  job  aids  as  consistent  or  totally  consistent  with  Army  doctrine.  Even  though  the 
BBSE  was  the  newest  of  the  FXXITP  products,  these  results  suggest  that  the  lag  time  between 
development  and  field  implementation  of  structured  TSPs  is  sufficient  to  introduce  discrepancies 
as  Army  doctrine  continues  to  evolve. 

Only  8  of  28  training  audience  members  (29%)  who  responded  to  the  questionnaire  reported 
that  they  used  their  Training  Audience  Guides  (see  Table  43).  Of  those  eight,  half  reported  that 
it  was  easy  to  find  information  in  the  guides,  and  half  said  it  was  neither  easy  nor  difficult. 
Several  of  them  commented  that  they  did  not  have  time  to  read  all  the  TSP  materials.  Other 
participants  (i.e.,  observers,  roleplayers,  interactors,  EXCON  personnel)  were  more  likely  to 
report  reading  their  guides:  Of  36  respondents,  26  (72%)  reported  that  they  used  their  respective 
guides,  and  22  (85%  of  those  who  used  their  guides)  said  that  finding  information  in  the  guides 
was  easy  or  very  easy. 
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Table  43 

Perceptions  of  Usefulness  of  Brigade  and  Battalion  Staff  Exercise  Materials 


Respondent 

Difficult  or 
Very  Difficult 

Neither 

Easy  or 
Very  Easy 

Did  not 

use 

N 

How  easy  to  find  information  in  guides  by 
. .  .Brigade  Training  Audience 

0 

2  (20%) 

2  (20%) 

6  (60%) 

10 

. .  .Battalion  Training  Audience 

0 

2(11%) 

2(11%) 

14  (78%) 

18 

...Observers 

0 

1  (20%) 

3  (60%) 

1  (20%) 

5 

. .  .Roleplayers  and  Interactors 

0 

1  (8%) 

5  (42%) 

6  (50%) 

12 

. .  .EXCON  Personnel 

0 

2  (10%) 

14  (74%) 

3  (16%) 

19 

Note.  EXCON  =  exercise  control. 


At  the  procedural  level,  three-fourths  of  the  EXCON  personnel  found  it  easy  or  very  easy  to 
send  scripted  messages  and  monitor  OPFOR  activities,  while  half  to  two-thirds  reported  the  same 
for  issuing  and  briefing  orders,  answering  the  brigade  staffs  questions,  moving  division/corps 
assets;  and  controlling  division/corps  CSS  functions.  Only  about  half  of  the  EXCON  personnel 
reported  attending  BBS  training,  as  the  TSP  recommended.  The  great  majority  of  them  felt  BBS 
training  was  essential,  with  75%  recommending  (to  a  moderate  extent  or  greater)  basic  or 
additional  BBS  training. 

Comments  of  the  participants  pointed  to  a  number  of  improvements  in  the  tactical  materials 
of  the  BBSE  TSP.  These  merit  consideration  in  future  efforts  to  improve  and  expand  the 
FXXITP  utility  and  impact.  The  recommendations  are  summarized  in  Table  44. 

Table  44 

Summary  of  Recommended  Brigade  and  Battalion  Staff  Exercise  Enhancements 


Area 

Recommendation 
Tactical  Materials 

•  Scrub  the  Intelligence  Annex  to  accurately  reflect  current  division  capabilities. 

•  Revamp  the  initial  intelligence  estimate  to  provide  more  specific  information  and  eliminate 
inconsistencies  (e.g.,  early  warning  priorities  should  match  division  collection  capabilities). 

•  Add  personnel  annexes  to  the  family  of  OPORDs. 

•  Expand  the  sketch  map  for  the  first  mission  into  a  full  Situation  Template. 

•  Build  greater  diversity  into  the  combat  mission  dynamics.  For  example,  avoid  the  BLUFOR  always 
attacking  through  the  most  restrictive  terrain. 

•  Improve  the  Intelligence  Summaries  by  making  them  more  regular  (e.g.,  every  six  hours)  and 
incorporating  actual  exercise  developments  in  them. 

•  Modify  the  timeline  for  the  reconnaissance  battle  to  make  it  more  realistic. 

•  Expand  the  TSP  materials  for  engineer  elements. 

•  Add  Army  Airspace  Command  and  Control  (A2C2)  materials,  to  include  an  A2C2  annex  and 
overlay  as  well  as  appropriate  job  aids  (checklists,  planning  guides,  etc.). 

table  continues 
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Table  44  (continued) 
Area 


_ Recommendation _ 

•  Improve  the  division  air  defense  artillery  (ADA)  Annex  to  make  it  mission-focused,  to  include 
high/medium  range  air  defense  coverage  parameters. 

•  Expand  the  number  of  division-level  events  requiring  the  brigade  to  process  and  analyze 
information. 

•  Improve  the  replication  of  U.S.  Air  Force  processes  and  products. 

•  Strongly  encourage  communication  channels  linking  ADA  assets  with  brigade  and  task  force  TOCs. 
Performance  Assessment  and  Feedback  Procedures 

•  Provide  focused  training  modules  for  observers  to  ensure  more  comprehensive,  consistent 
performance  measurement  and  AARs. 

•  Include  instructions  for  coaching  as  well  as  desired  AAR  procedures. 

•  Put  the  observers  on  a  rotating  schedule  inside  the  CPs  to  help  cut  down  the  CP  traffic,  but  sufficient 
observation  opportunities  would  need  to  be  protected. 

•  Include  the  battlestaff  NCOs  in  the  AARs  even  though  they  are  not  part  of  the  formal  training 
audience. 

•  Provide  a  take  home  package  to  provide  the  unit  a  compendium  of  feedback  to  study  later. 

•  Give  observers  an  administrative  net  (via  hand-held  radios)  to  share  information  and  coordinate 
activities. 

Procedures  for  Roleplayers  and  Interactors 

•  Provide  explicit  TSP  instructions  regarding  shift  change  briefing  procedures  for  the  benefit  of 
personnel  manning  the  simulation  workstations.  Some  BLUFOR  roleplayers  and  interactors 
indicated  they  experienced  confusion  following  shift  changes.  While  these  procedures  should  be 
contained  in  the  unit’s  current  standing  operating  procedures  (SOP),  the  simulation  exercises  are  just 
foreign  enough  to  warrant  more  explicit  guidance. 

Exercise  Control  (EXCON)  Guidance 

•  Provide  a  high-level  exercise  overview,  including  the  TOC-Battle  Simulation  Center  linkage  and  the 
BBS  implementation  concept. 

•  Include  call  signs  for  out-of-sector  units. 

•  Use  the  NTC  Standing  Operating  Procedure  (SOP)  for  reporting. 

•  Provide  an  integrated  matrix  listing  BLUFOR  events,  linked  to  scripted  message  numbers. 

•  Include  a  practical  exercise  as  part  of  their  pre-exercise  train-up. 

Performance  Objective  Additions 

•  Include  RSOI  requirements.  The  first  mission,  for  example,  could  begin  with  RSOI  tasks  that 

would  transition  to  force-on-force  activities.  The  RSOI  activities  would  emphasize  building  combat 
power,  establishing  the  base  for  definitive  combat  action.  High-value  tasks  would  include  drawing 
combat  equipment,  assessing  the  status  of  equipment  and  personnel,  organizing  the  staging  base, 
establishing  CPs,  and  maintaining  reception  area  security. _ 

Note.  OPORD  =  operation  order;  BLUFOR  =  Blue  Forces;  TSP  =  training  support  package;  ADA  =  air  defense 
artillery;  TOC  =  tactical  operation  center;  AAR  =  after  action  review;  CP  =  command  post;  NCO  = 
noncommissioned  officer;  BBS  =  Brigade/Battalion  Battle  Simulation;  SOP  =  standing  operating  procedure;  RSOI  = 
reception,  staging,  onward  movement,  and  integration. 
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Brigade  and  Battalion  Staff  Exercise  Impact 


As  seen  in  Table  45,  nearly  all  (90%)  of  the  brigade  respondents  felt  (to  a  great  or  very  great 
extent)  that  the  exercise  enabled  them  to  practice  techniques  they  would  use  in  battle.  The 
corresponding  figure  for  the  secondary  training  audience  was  somewhat  lower  (63%)  but  only 
one  respondent  was  negative  (5%).  Additionally,  the  majority  (80%  or  more)  of  the  brigade- 
level  and  task  force-level  training  audiences  perceived  that  their  individual  understanding  and 
abilities  had  improved  to  a  moderate  extent  or  greater.  It  is  notable  that  100%  of  both  training 
audience  samples  felt  their  own  teamwork  skills  had  improved  at  least  moderately,  with  84%  or 
more  indicating  this  specific  improvement  to  a  great  or  very  great  extent.  These  combined 
results  indicate  that  the  bulk  of  the  training  audience  perceived  the  relevance  of  the  BBSE 
training. 


Table  45 

Perceptions  of  Impact  of  the  Brigade  and  Battalion  Staff  Exercise  on  Individual  Users 


Respondent 

Area  of  Impact 

No  Extent  or 
Slight  Extent 

Moderate 

Extent 

Great  Extent  or 
Very  Great  Extent 

N 

Brigade  Training  Audience 

Enabled  practice  on  battle  techniques 

0 

1  (10%) 

9  (90%) 

10 

Improved  understanding  of  own  tasks 

1  (10%) 

3  (30%) 

6  (60%) 

10 

Improved  ability  to  perform  own  tasks 

1  (10%) 

2  (20%) 

7  (70%) 

10 

Improved  own  teamwork  skills 

0 

1  (10%) 

9  (90%) 

10 

Task  Force  Training  Audience 

Enabled  practice  on  battle  techniques 

1  (5%) 

6  (32%) 

12  (63%) 

19 

Improved  understanding  of  own  tasks 

2(11%) 

9  (47%) 

8  (42%) 

19 

Improved  ability  to  perform  own  tasks 

2(11%) 

9  (47%) 

8  (42%) 

19 

Improved  own  teamwork  skills 

0 

3(16%) 

16  (84%) 

19 

Observers 

Improved  brigade  staff  proficiency 

1  (25%) 

0 

3  (75%) 

4 

Improved  task  force  staff  proficiency 

1  (25%) 

1  (25%) 

2  (50%) 

4 

Improved  staff  members’  teamwork  skills 

0 

2  (40%) 

3  (60%) 

5 

As  Table  45  shows,  the  external  observers’  perceptions  of  performance-related 
improvements  reinforced  those  of  the  training  audience.  Three  of  the  four  respondents  felt  that 
staff  proficiency  had  improved,  and  four  of  five  respondents  reported  improvements  in  staff 
teamwork  skills. 

As  shown  in  Table  46,  all  of  the  brigade-level  training  audience  and  nearly  all  of  the  task 
force-level  training  audience  perceived  that  the  BBSE  helped  meet  unit  training  needs  to  a 
moderate  extent  or  greater.  This  included  enhancing  combat  readiness. 
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Table  46 

Perceptions  of  Impact  of  the  Brigade  and  Battalion  Staff  Exercise  on  the  Unit 


Respondent 

Area  of  Impact 

No  Extent  or 
Slight  Extent 

Moderate 

Extent 

Great  Extent  or 
Very  Great  Extent 

N 

Brigade  Training  Audience 

Help  meet  staff  training  goals 

0 

3  (30%) 

7  (70%) 

10 

Pave  way  for  follow-on  training 

0 

3  (30%) 

7  (70%) 

10 

Better  prepare  unit  for  National 

0 

2  (20%) 

8  (80%) 

10 

Training  Center 

Enhance  unit  combat  readiness 

0 

3  (30%) 

7  (70%) 

10 

Save  unit  training  time 

1  (10%) 

3  (30%) 

6  (60%) 

10 

Save  unit  training  funds 

1  (10%) 

4  (40%) 

5  (50%) 

10 

Task  Force  Training  Audience 

Help  meet  staff  training  goals 

0 

7  (37%) 

12  (63%) 

19 

Pave  way  for  follow-on  training 

1  (5%) 

4  (21%) 

14  (74%) 

19 

Better  prepare  unit  for  National 

1  (5%) 

4(21%) 

14  (74%) 

19 

Training  Center 

Enhance  unit  combat  readiness 

3  (16%) 

5  (26%) 

11(58%) 

19 

Save  unit  training  time 

5  (26%) 

5  (26%) 

9  (47%) 

19 

Save  unit  training  funds 

4(21%) 

3  (16%) 

12(63%) 

19 

As  Table  47  shows,  75%  or  more  of  both  training  audience  samples  estimated  the  exercise 
saved  their  unit  substantial  training  time  and  funds,  with  benefits  solidly  outweighing  costs. 
Additionally,  all  of  the  brigade  staff  respondents  and  nearly  all  of  the  task  force  respondents 
believed  (to  a  moderate  extent  or  greater)  that  their  own  unit  should  use  the  BBSE  for  future 
training  and  that  other  units  should  use  the  exercise  as  well.  The  external  observers’  responses 
strongly  reinforced  those  of  the  training  audience.  Thus,  the  actual  and  potential  contributions  of 
the  BBSE  to  the  unit’s  training  program  received  quite  favorable  marks. 

After  the  NTC,  members  of  the  training  audience  were  asked  for  their  perceptions  of  the 
overall  impact  of  the  BBSE  on  their  NTC  performance.  The  responses  on  questions  of  whether 
their  own  or  other  units  should  use  the  BBSE  were  generally  positive,  and  more  positive  among 
brigade  staff  respondents  than  task  force  respondents  (Table  48). 
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Table  47 

Perceptions  of  the  Future  Utility  of  Brigade  and  Battalion  Staff  Exercise 


Respondent 

No  Extent  or 

Moderate 

Great  Extent  or 

Who  Should  Use 

Slight  Extent 

Extent 

Very  Great  Extent 

N 

Brigade  Staff 

Benefits  outweigh  costs 

0 

4  (44%) 

5  (56%) 

9 

Own  brigade  should  use 

0 

3  (30%) 

7  (70%) 

10 

Other  brigades  should  use 

0 

3  (30%) 

7  (70%) 

10 

Task  Force  Staff 

Benefits  outweigh  costs 

1  (6%) 

5  (29%) 

1 1  (65%) 

17 

Own  task  force  should  use 

2(11%) 

5  (28%) 

11  (61%) 

18 

Other  task  force  should  use 

2(11%) 

3(17%) 

13  (72%) 

18 

Observers 

This  brigade  should  use 

0 

3  (60%) 

2  (40%) 

5 

Other  brigades  should  use 

0 

2  (40%) 

3  (60%) 

5 

Table  48 

Perceptions  of  the  Future  Utility  of  the  Brigade  and  Battalion  Staff  Exercise  for  Sustaining 

Proficiency 

Measure 

No  Extent  or 

Moderate 

Great  Extent  or 

Respondent 

Slight  Extent 

Extent 

Very  Great  Extent 

N 

Own  brigade  or  task  force  should  use  to 
sustain  proficiency 

Brigade  Staff 

0 

1  (25%) 

3  (75%) 

4 

Task  force  Staff 

0 

5  (62%) 

3  (38%) 

8 

Other  brigades  or  task  forces  should  use 
for  National  Training  Center  Preparation 

Brigade  staff 

0 

1  (25%) 

3  (75%) 

4 

Task  forces  staff 

0 

6  (75%) 

2  (25%) 

8 

As  shown  in  Table  49,  despite  the  low  response  rates,  the  ratings  of  the  benefits  of  BBSE 
participation  in  terms  of  NTC  performance  were  entirely  positive  from  the  brigade-level  training 
audience,  and  somewhat  lukewarm  from  the  task  force  respondents. 
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Table  49 

Perception  of  the  Impact  of  the  Brigade  and  Battalion  Staff  Exercise  on  National  Training  Center 
Performance 


Respondent 

Harmful  or 
Very  Harmful 

Neither 

Beneficial  or 
Very  Beneficial 

N 

Brigade  staff 

0 

0 

4(100%) 

4 

Task  force  staff 

0 

3  (43%) 

4  (57%) 

7 

Brigade  and  Battalion  Staff  Exercise  Supportabilitv 

There  were  quite  a  few  observations  from  surge  team  members  and  comments  from  BBSE 
participants  that  address  the  supportability  of  the  BBSE.  They  concern  personnel,  time,  and  T3 
issues.  The  observations  and  comments  are  summarized  below. 

The  BBSE  TSP  recommends  that  brigades  designate  a  Combined  Arms  Operations  at 
Brigade  Level,  Realistically  Achieved  Through  Simulation  (COBRAS)  Coordinator  to  plan  and 
manage  the  preparation  for  each  exercise.  Both  brigades  participating  in  this  assessment  tasked 
an  Assistant  S3  to  serve  as  their  coordinator  for  the  preparation/execution  of  the  BBSE.  With 
numerous  other  duties,  these  key  support  personnel  found  it  difficult  to  devote  sufficient  time  to 
coordinating  all  of  the  BBSE  preparation  activities.  In  addition,  the  rapid  start-up  of  the  ISAT 
project  constrained  the  planning  environment,  giving  the  units  little  lead  time  to  adjust  training 
calendars  that  had  been  established  months  before.  The  combination  of  time  constraints  and 
competing  demands  left  the  unit  coordinators  with  minimal  time  to  become  thoroughly  familiar 
with  all  of  the  BBSE  materials  and  make  informed  decisions  about  how  to  use  them  to  meet  unit 
training  needs.  This  situation  often  led  to  last-minute  actions  and  abbreviated  preparations  for 
collective  events,  in  effect  detracting  from  the  full  training  value  achievable  by  proper  use  of  the 
various  products. 

To  maximize  the  utilization  of  the  products,  ISAT  provided  support  personnel  to  assist  the 
participating  units,  as  discussed  in  Section  2.  The  on-site  TSCs  and  surge  teams  played  key  roles 
throughout  the  course  of  the  assessment.  In  addition,  BBSE  developers  made  up  the  surge  team 
that  conducted  start-up  training.  The  anecdotal  evidence  indicates  strongly  that  the  external 
support  was  critical  to  the  limited  implementation  that  was  possible. 

For  each  participating  brigade,  ISAT  members  conducted  all  start-up  training  (FXXITP 
orientation,  product  familiarization,  and  T3  sessions)  in  this  limited  implementation.  For 
Brigade  C,  for  example,  the  following  on-site  schedule  was  executed:  product  overview,  40 
minutes;  BSTS  overview,  20  minutes;  and  vignette  orientation  and  workshop,  55  minutes.  Team 
members  felt  these  were  minimum  essential  times,  and  recommended  additional  time  for  the 
BSTS  overview  whenever  BSTS  orientation  and  hands-on  training  are  scheduled  separately. 

The  ISAT  members  were  able  to  use  outline  and  briefing  slide  materials  successfully  because 
they  were  thoroughly  familiar  with  the  products  as  a  result  of  working  on  the  development 
teams.  However,  their  feeling  was  that  unit  personnel  (e.g.,  the  COBRAS  Coordinator)  would 
not  have  been  able  to  conduct  this  training  without  an  expanded  program  of  instruction,  to 
include  scripts  and  audiovisual  materials. 
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Duplication  and  distribution  of  training  materials  for  the  BBSE  in  this  assessment  fell  short 
of  the  scheme  called  for  in  the  TSP.  Several  features  of  the  training  environment  (e.g.,  time 
constraints,  competing  demands,  over-committed  personnel)  explain  most  of  the  difficulties  in 
distributing  essential  materials.  The  ubiquitous  nature  of  these  environmental  features  points  to  a 
pressing  need  to  explore  alternative  means  for  distributing  normally  printed  training  materials. 

For  the  BBSE,  the  TSPs  call  for  key  support  personnel,  to  include  the  Exercise  Director, 
observers,  the  OPFOR  Controller,  BLUFOR  roleplayers  and  interactors,  and  EXCON.  In 
contrast  to  the  vignettes  and  the  Janus  SIMEX,  these  roles  were,  for  the  most  part,  filled  by 
appropriate  personnel  from  the  division  and  a  sister  brigade.  As  a  result,  the  BBSE  was 
successfully  executed,  with  positive  comments  and  questionnaire  responses  from  the  audience 
and  observers. 


Summary 

This  section  has  presented  results  of  the  data  collection  efforts  throughout  the  course  of  the 
project.  In  addition  to  continuous  observations  of  units’  efforts  to  utilize  the  FXXITP  products 
(often  in  the  face  of  competing  demands  on  resources),  there  were  several  formal  data  collection 
opportunities.  At  each  of  these  opportunities,  we  attempted  to  assess  attitudes  concerning  all  the 
products  that  had  been  used  earlier.  The  result  was  a  long  retrospective  look  at  the  perceived 
utility  of  the  products. 

However,  because  only  one  brigade  was  able  to  follow  through  with  use  of  most  of  the 
products,  the  number  of  respondents  to  questionnaires  was  low.  Another  situation  that  affected 
the  amount  of  data  was  the  loss  of  BSTS  data  due  to  technical  difficulties. 

Nonetheless,  the  data  reported  here  reveal  some  general  trends  in  reactions  to  use  of  the 
product.  Those  trends  are  discussed  in  terms  of  lessons  learned  in  the  following  section. 

Section  5.  Lessons  Learned 

The  work  performed  during  the  ISAT  project  revealed  a  wide  range  of  issues  concerning  the 
implementation  of  the  FXXITP  products  as  well  as  the  support  and  assessment  conducted  by  the 
project  team.  This  section  discusses  those  issues  from  the  perspective  of  lessons  learned  during 
the  project  and  recommendations  that  emerged  in  discussions  with  members  of  the  participating 
units.  As  were  the  results,  the  lessons  are  grouped  in  three  areas:  acceptability  (including 
usability),  perceptions  of  impact,  and  supportability.  A  final  lesson  addresses  training 
effectiveness  assessment  itself. 


Acceptability 

During  this  assessment  acceptability  was  defined  as  “...the  unit’s  acceptance  of  the  FXXITP 
products  as  being  doctrinally  correct  and  containing  usable  materials.”  Questions  that  were 
addressed  included:  Were  the  training  materials  doctrinally  correct?  Were  they  usable?  Were 
other  materials  necessary? 
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The  acceptability  of  the  FXXITP  products  was,  for  the  most  part,  positive.  BSTS  users 
indicated  that  the  BSTS  met  the  basic  needs  for  individual  staff  level  training,  although  it  was 
not  sufficiently  interactive.  The  vignettes  were  perceived  as  sufficiently  flexible  for  use  in 
learning  centers,  in  the  field,  or  for  “Thursday  morning”  (staff  development)  training.  The  BSTS 
and  BSE  were  found  by  7ATC  reviewers  to  be  the  least  current  of  the  products.  Most  felt  that 
the  products  supported  the  training  they  were  intended  to  support.  Yet  many  comments 
indicated  that  there  were  changes  that  needed  to  be  made  to  make  the  products  more  useful  and 
keep  them  viable. 

Lesson:  Structured  training  products  will  inevitably  become  outdated,  and  will  require 
continual  examination  and  updating. 

It  took  over  five  years  to  develop,  implement,  and  assess  the  FXXITP  products.  Because 
Army  doctrine  has  continued  to  evolve,  some  of  the  products  are  currently  outdated,  doctrinally 
and  technologically.  As  stated  in  Section  4,  most  of  the  shortcomings  of  the  FXXITP  products 
deal  with  doctrinal  changes  to  terms  and  symbols,  with  some  changes  to  tactics,  techniques,  and 
procedures,  and  newer  ways  to  deliver  information.  This  is  an  inevitable  occurrence  for  any 
structured  training  product,  and  underlies  one  of  the  reasons  that  the  program  must  have  Army 
wide  support  (as  described  in  a  later  lesson). 

Lesson:  Even  though  it  is  inevitable  that  the  products  will  become  outdated,  they  can 
be  used  while  updating  goes  on. 

Despite  the  doctrinal  shortcomings  of  the  products,  the  units  were  still  able  to  use  the 
products  for  training,  and  after  their  LTP,  indicated  that  the  training  had  been  useful.  For 
example,  reviewers  of  the  BSTS  from  the  project,  user  units,  and  7ATC  identified  many 
instances  of  outdated  doctrine;  yet  users  of  BSTS  were  consistently  positive  regarding  the 
doctrinal  currency  of  the  BSTS  courses. 

When  asked  about  the  apparent  discrepancy,  one  reviewer  said  that  the  fighting  units  were 
more  likely  to  overlook  doctrinal  inconsistencies  if  the  potential  training  benefit  was  high.  In 
other  words,  they  cared  more  about  learning  to  fight  well  than  how  they  learned  how  to  fight 
well.  Perhaps  these  units,  who  train  and  are  prepared  to  fight,  are  less  concerned  about 
meticulous  consistency  with  all  the  points  of  doctrine  than  are  people  involved  in  the 
development  and  teaching  of  doctrine.  That  is,  when  Brigade  C  commented  on  the  value  of  the 
BSTS,  they  were  more  interested  in  the  potential  benefit  the  training  could  offer  than  the  small 
discrepancies  in  doctrine.  They  tended  to  overlook  doctrinal  inconsistencies. 

While  doctrinal  consistency  is  critical,  it  may  be  that  intermittent  updates,  and  not  a 
continuously  ongoing  revision  process,  are  good  enough.  The  fact  is  that  (as  stated  in  the 
previous  lesson)  all  structured  training  products  that  are  tied  to  doctrine  will  become  outdated  as 
doctrine  changes.  But  the  Army,  or  TRADOC,  or  the  FXXITP,  should  find  a  way  to  distinguish 
between  doctrinal  inconsistencies  that  lead  to  bad  training,  and  those  that  are  below  the  noise 
level.  They  should  avoid  the  situation  where  any  and  all  instances  of  doctrinal  outdated  graphics 
or  language  can  cause  the  distribution  and  usage  of  products  to  be  delayed.  It  seems  apparent 
that  training  units  are  willing  and  able  to  overlook  flaws  of  products  if  those  products  are 
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perceived  to  be  good  training  products.  One  comment  from  a  BBSE  participant  indicates  that 
doctrinal  currency  may  not  be  paramount:  “[It  is  doctrinal],  but  not  what  the  NTC  did  to  me 
when  we  were  there  six  months  ago.” 


The  thrust  of  the  participant  reactions  is  that,  in  the  short  run,  the  FXXITP  products  can 
meet  the  training  needs  despite  their  doctrinal  shortcomings.  However,  units  will  quickly 
become  tired  of  training  with  outdated  products  and  will  either  quit  using  them  or  be  required  to 
update  the  products  themselves.  If  the  units  quit  using  them  or  have  to  update  the  products 
t  emselves  the  Army  will  have  lost  all  of  the  advantages  of  standardized  and  structured  training 
o  ensure  that  these  products  remain  viable  to  the  unit  while  remaining  standardized  and 
structured,  program  resources  should  be  allocated  to  gather  feedback  on  these  products,  review 

them  for  currency,  and  update  them  in  a  timely  manner.  The  force  should  be  kept  informed  of 
the  updating  intentions  and  progress. 

One  way  to  facilitate  the  maintenance  of  doctrinal  currency  would  be  to  use  the  same 
products  for  institutional  training.  By  doing  so,  the  training  institutions  (who  are  also 
responsible  for  updating  doctrine)  will  by  the  nature  of  their  institutional  training  requirements 
focus  on  product  currency.  This  also  provides  an  advantage  of  more  widespread  understanding 
o  e  products,  since  unit  leaders  will  have  used  them  during  their  institutional  training. 

Lesson:  Future  TSPs  can  be  more  usable  if  we  incorporate  more  of  the  human 
dimensions  of  TSP  usage. 

A  consistent  concern  expressed  about  use  of  the  FXXITP  products  was  the  size  of  the  TSPs 
Because  of  the  intricate  structures  of  the  programs,  the  TSPs  for  all  of  the  products  are  lengthy 
and  require  some  practice  to  navigate.  Even  with  TSCs,  surge  team  assistance,  and  T3  sessions, 
the  sheer  appearance  of  complexity  was  daunting.  Many  participants  indicated  that  they  did  not 
use  their  materials.  Unit  trainers  commented  that  it  was  difficult  to  reproduce,  distribute,  and 
explain  the  contents  of  the  materials.  The  data  (or  rather,  the  lack  of  data)  indicated  clearly  the 
preva  ence  of  problems  with  training  management  components  that  often  left  unit  leaders  only 

aWare  of  Product  options  and  value,  and  lacked  safeguards  to  prevent  loss  of  data 
(BSTS  courses). 


Such  fmdmgs  indicate  the  products  did  not  fully  meet  the  working  needs  of  the  training 
audience.  One  likely  cause  of  this  circumstance  lies  in  the  nature  of  the  requirements  analysis 
that  is  performed  when  structured  training  products  are  developed.  Developers  have  not  yet 
identified  effective  ways  to  address  training  constraints  with  the  same  expertise  that  is  applied  to 
task-based  training  needs.  The  design-oriented  phases  focus  heavily  on  product  content,  scenario 
context,  and  simulation  characteristics.  However,  developers  have  not  yet  been  able  to 
incorporate  what  is  known  about  knowledge  engineering  and  the  constraints  of  the  training 
environment.  Adjusting  the  design  process  to  bring  the  human  dimensions  of  TSP  usage  into 
closer  balance  with  training  objectives  and  technology  considerations  could  lead  to  products  that 
make  the  information  easier  for  the  users  to  access  and  reduce  the  burden  of  getting  ready  to 
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The  methodologies  that  emerged  from  early  structured  training  development  research  (C.  H. 
Campbell,  Campbell,  Sanders,  Flynn,  &  Myers,  1995;  C.  H.  Campbell  &  Deter,  1997;  C.  H. 
Campbell,  Ford,  Campbell,  &  Quinkert,  1998)  call  for  analysis  of  staffing  requirements,  training 
audience  characteristics,  training  environment  characteristics,  and  human  factors  aspects  of 
training  products.  As  yet,  no  controlled  study  of  the  effectiveness  of  different  TSP  approaches 
has  been  attempted.  Such  considerations  may  provide  a  substantial  payoff  to  both  training 
developers  and  researchers. 

This  may  indicate  a  need  to  reexamine  the  entire  implementation  model.  It  is  easy  to  say 
that  we  must  streamline  the  presentation  of  materials  and  minimize  the  load  imposed  on  the 
trainers  and  training  audience.  At  the  same  time,  it  is  imperative  that  crucial  information  (e.g., 
training  objectives,  definition  of  training  audience  versus  support  personnel,  specific 
assignments)  be  disseminated  in  advance,  and  that  all  necessary  information  and  guidance  that 
defines  the  product  structure  be  available.  Failure  to  provide  clear  information  in  advance 
inevitably  leads  to  confusion  about  the  exercise’s  purpose,  roles  and  duties,  which  in  turn 
degrades  the  event’s  training  value.  Failure  to  provide  materials  that  guide  the  conduct  of  the 
exercise  causes  exercise  support  personnel  to  make  their  best  guess  about  how  to  allow  events  to 
unfold,  which  breaks  the  link  between  the  training  and  the  tasks  that  are  to  be  trained. 

Developers  of  structured  training  products  have  expressed  frustration  with  the  lack  of 
complete  use  of  TSP  materials  for  years  (C.  H.  Campbell,  Graves,  et  al.,  1998;  C.  H.  Campbell  et 
al.,  1999;  Graves  et  al.,  1997).  Each  new  development  attempts  to  provide  for  more  user- 
friendly  TSPs,  yet  we  have  not  yet  been  able  to  crack  the  code  on  user  needs.  We  have  simply 
not  been  able  to  comprehend  the  obstacles  and  incorporate  effective  solutions.  The  answer  is  not 
a  simple  one.  It  is  likely  to  be  a  combination  of  additional  research  on  what  works  for  time- 
stressed  units,  Army  and  command  emphasis  on  use  of  the  products,  and  the  availability  of  surge 
teams  or  support  coordinators. 

Lesson:  There  needs  to  be  a  balance  between  the  time  allotted  to  prepare  for  these 
exercises  and  the  time  it  takes  to  prepare. 

These  FXXITP  products  are  not  advertised  as  “turn-key”  training,  which  would  convey  a 
design  characteristic  that  is  supposed  to  enable  the  training  coordinators  to  implement  the 
training  products  with  minimal  preparation.  Nonetheless,  unit  training  coordinators  rarely 
allocate  sufficient  time  for  preparation.  Time  has  proven  to  be  the  most  valuable  commodity  to 
the  units  participating  in  this  assessment,  and  is  probably  equally  valuable  across  AC  and  RC 
units.  As  a  general  rule,  personnel  did  not  study  the  guides  or  accomplish  preparation  activities 
in  accordance  with  the  TSP  instructions.  It  is  easy  for  us  to  blame  the  TSPs  when  in  fact  it  may 
be  that  the  unit  isn’t  providing  enough  time  to  prepare  for  training. 

Lesson:  In  addition  to  doctrinal  updates,  TRADOC  and  the  FXXITP  should  continue  to 
incorporate  technology  updates. 

This  lesson  emerged  as  the  team  observed  BSTS  and  BBSE  implementation.  Units  in  this 
assessment  used  BSTS  courseware  in  stand-alone  mode.  However,  the  courses  were  designed 
primarily  for  a  LAN  operating  environment.  This  is  not  the  LAN  environment  we  are  used  to 
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using  today,  but  a  hard- wired  Ethernet  LAN  that  was  commonplace  in  1993.  Similarly,  the 
BBSE  was  originally  written  for  a  version  of  BBS  software  that  was  no  longer  being  used  at  the 
time  of  implementation.  The  IS  AT  team  assisted  the  simulation  center  personnel  with  the 
updating  to  allow  the  exercise  to  occur.  This  shows  a  need  to  validate  assumptions  about 
infrastructure  in  place  when  products  are  ready  for  fielding.  Although  a  comprehensive  systems 
approach  builds  infrastructure  in  parallel  with  course  development,  the  technology  needs  to  be 
reexamined  continually,  just  as  do  doctrinal  foundations. 

Lesson:  TRADOC  and  the  FXXITP  should  continue  to  develop  ways  for  units  to  tailor 
TSP  materials  to  their  unit’s  organization  and  equipment,  training  audience, 
and  SOP. 

This  lesson  addresses  a  range  of  issues  that  attack  the  advantages  of  structure  and 
standardization,  balanced  against  the  need  for  programs  that  are  flexible  enough  for  valuable 
training.  Allowing  units  to  tailor  tactical  materials  is  relatively  simple  in  the  vignettes,  but 
becomes  more  complex  in  the  BSE  and  BBSE,  and  extremely  difficult  in  the  BSTS.  Except  for 
the  BSTS,  TSPs  could  be  provided  in  an  electronic  form  so  that  the  unit  can  modify  the  unit 
information  to  match  their  own.  Another  option  would  be  to  deliver  these  TSPs  in  a  manner 
consistent  with  the  Commander’s  Integrated  Training  Tool  (Gossman  et  al.,  1999)  This 
computer-  and  Internet-based  system,  currently  under  development  by  ARI  and  Simulation 
Training  and  Instrumentation  Command,  is  designed  to  help  units  use  existing  TSPs  as  written, 
modify  them  for  their  own  situations,  or  develop  new  TSPs. 

The  current  approach  of  designing  structured  TSPs  to  be  used  by  a  diverse  mix  of  tactical 
units  makes  development  a  viable  process,  but  the  one-size-fits-all  approach  asks  unit  personnel 
to  adopt  alter-identities,  notional  equipment  mixes,  and  so  on.  On  the  other  hand,  the  desire  to 
replicate  exactly  the  unit’s  organization,  equipment,  signal  operations  instructions,  tactical  SOP, 
and  other  operational  conditions  is  understandable,  but  there  are  no  data  showing  that  precise 
replication  increases  training  value  or  transfer  of  training.  Research  to  evaluate  this  dimension, 
as  well  as  to  find  ways  to  make  the  products  more  flexible,  would  help  determine  if  the  Army 
should  invest  in  unit-specific  TSPs. 

In  addition  to  the  desire  to  tailor  the  organizational  information,  the  units  participating  in 
this  assessment  consistently  modified  preparation  and  execution  procedures  specified  in  the 
FXXITP  TSPs.  Abbreviated  preparation  activities,  training  audience  changes,  altered  missions, 
scheduling  peculiarities,  self-generated  performance  assessment  and  AAR  procedures,  and 
numerous  other  alterations  were  common.  This  indicates  that,  even  with  structured  training 
products,  standardization  is  difficult  to  achieve.  More  to  the  point,  it  highlights  the  importance 
of  designing  TSPs  to  provide  substantial  execution  flexibility.  For  example,  the  vignette  training 
audience  was  typically  expanded  by  the  unit,  but  the  “extra”  participants  found  no  specific  TSP 
materials  for  themselves.  The  TSP  could  include  instructions  for  integrating  extra  participants 
and  outfitting  them  with  relevant  training  materials.5  The  design  process  should  take  into 


5  The  reader  should  notice  at  this  point  the  insidious  subtlety  of  “good  idea  creep;”  every  such  good  idea 
increases  the  bulk  of  the  TSP,  and  we  have  yet  to  discover  how  to  include  the  many  good  ideas  without 
complicating  TSP  delivery  even  further. 
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account  the  likely  variations  in  utilization,  and  the  evaluation  process  should  incorporate 
program  flexibility  as  a  major  dimension  of  interest. 

The  single  implementation  of  the  BBSE  is  a  case  in  point.  The  product  was  altered 
significantly  to  meet  the  unit’s  perceived  training  needs  in  preparation  for  an  NTC  rotation.  Yet 
the  BBSE  exercise  director,  the  installation’s  deputy  commanding  general,  pointed  out  that  “If 
we  had  to  generate  this  exercise  on  our  own,  given  our  organization  and  other  requirements,  we 
would  not  have  done  as  well.  Overall,  I  think  the  content  of  this  TSP  is  very  good,  the  concept  is 
good.” 

Perceptions  of  Impact 

Impact,  in  this  research,  included  issues  of  learning,  job  performance,  and  organizational 
impact.  Questions  of  interest  included:  Were  the  training  products  useful  for  learning  and 
practicing  job  requirements  and  preparing  for  other  major  training  events?  Did  users  think  that 
the  training  had  (or  would  have)  a  positive  effect  on  performance?  Most  of  the  responses  from 
users  indicated  that  they  saw  the  potential  training  value  of  the  products.  After  the  LTP 
experience,  the  Brigade  C  members  said  that  the  products  had  helped  them  prepare;  after  the 
NTC  rotation,  their  comments  showed  that  they  were  beginning  to  see  some  value  in  the 
products.  Many  of  the  answers  to  these  questions  focused  less  on  the  existing  products  than  they 
did  on  what  was  still  needed — a  subtle  tribute  to  the  perceived  training  impact. 

Lesson:  The  existing  products  need  to  be  expanded. 

The  call  for  additional  expanded  products  came  mostly  on  the  BSTS  and  vignettes.  On  the 
BSTS,  users  requested  courses  for  their  staff  section  members  as  well  as  for  the  staff  officers. 
While  there  are  cases  where  staff  officers  have  used  the  BSTS  courses  for  their  sections,  specific 
courses  would  provide  the  information  and  training  that  NCOs  need. 

On  vignettes,  they  commented  that  brigade  staff  vignettes  are  not  enough.  The  units  also 
need  vignettes  for  battalion  staffs,  for  brigade  leadership,  for  leaders  linked  from  brigade  through 
company  and  for  commanders  and  their  staffs.  The  so-called  “Leader  Vignettes”  are  needed  to 
ensure  that  all  staff  section  leaders  and  commanders  within  the  brigade  combat  team  can  develop 
a  common  understanding  of  the  brigade  commander’s  intent  and  information  needs.  The 
multiechelon  leader  vignettes  would  help  the  subordinate  commanders  understand  the  amount 
and  type  of  detail  the  brigade  commander  needs  to  make  a  decision.  These  would  likely  be 
closely  associated  with  the  commander’s  critical  information  requirements  and  Decision  Support 
Template.  The  goal  of  these  vignettes  would  be  to  provide  a  concrete  experience  (adult  learning 
model  start  point)  from  which  the  participants  could  discuss  the  “art”  of  warfighting.  Similarly, 
the  commander-staff  vignettes  would  bring  the  key  leaders  of  the  staff  into  the  process  where  the 
staffs  add  clarity  to  the  information  provided  by  the  commanders  because  they  have  the  time  to 
analyze  the  information.  One  user  explained  that  the  outcomes  of  the  vignettes  for  leaders 
should  be  staff  recommendations  for  commanders;  the  vignette  should  train  staff  leaders  to 
analyze  the  right  amount  of  information  quickly. 
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Supportabilitv 


Supportability  of  the  products  referred  to  whether  a  brigade  would  be  able  to  conduct  the 
FXXITP  exercises  within  the  resources  available  to  them.  The  key  question  was:  What  would  it 
take  to  make  the  training  products  useable  within  a  brigade’s  training  plan?  In  this  regard  each 
component  of  the  FXXITP  had  to  be  evaluated  for  supportability  based  on  the  requirements 
established  for  its  use,  as  laid  out  in  the  TSP. 

The  experience  on  this  project  would  indicate  that  all  of  the  assessed  components  of  the 
FXXITP  require  external  support  for  unit  use.  The  amount  of  support  will  vary.  For  the  BSTS, 
both  system  installation  and  initial  use  training  were  provided,  and  were  later  supplemented  with 
continuing  assistance  in  recording  usage  data  and  keeping  the  systems  running.  The  support  for 
the  vignettes  was  relatively  slight,  involving  only  initial  training  for  the  unit  personnel.  For  the 
more  complex  elements,  the  Janus  SIMEX  (which  served  as  stand-in  for  the  BSE)  and  the 
BBSE,  the  project  surge  team  was  in  great  demand.  The  Janus  SIMEX  was  anomalous:  the 
project  team  prepared  the  TSP  with  full  expectation  of  providing  a  great  deal  of  implementation 
assistance.  But  the  BBSE,  with  its  relatively  comprehensive  TSP,  also  called  for  significant 
surge  team  efforts. 

The  five  lessons  stated  below  address  three  aspects  of  supportability. 

Lesson:  The  program  and  its  products  will  require  Army-wide  support  in  order  to  be 
resourced  and  used. 

One  of  the  biggest  concerns  of  the  personnel  using  the  products  was  the  level  of  support 
given  to  the  FXXITP  and  the  products  of  the  program  Army- wide.  Without  the  proper  emphasis 
from  the  Army  chain  of  command,  most  participants  felt  that  the  products  and  the  training  they 
were  intended  to  support  would  not  be  understood,  included  in  training  strategies  and  plans,  or 
resourced. 

In  order  for  the  FXXITP  to  be  institutionalized,  support  for  the  FXXITP  must  start  at  the 
DA  level  and  must  be  supported  by  each  major  command  that  will  use  the  products.  As  one 
brigade  commander  said,  “The  Force  XXI  Training  Program  must  be  endorsed  by  the  Army,  not 
just  by  individual  units.”  The  TRADOC  must  not  only  support  the  program  to  ensure  that 
personnel  attending  TRADOC  schools  understand  the  program  and  how  to  use  the  products, 
TRADOC  must  also  be  prepared  to  maintain  the  program  and  its  products  to  ensure  their 
continued  viability  and  usefulness  to  the  units  in  the  field.  The  FORSCOM  must  support  the 
program  by  establishing  a  training  strategy  that  helps  units  understand  how  to  incorporate  these 
products  into  their  training  plans,  and  must  also  incorporate  requirements  for  the  training  into 
FORSCOM  Regulations  350-1  and  350-2  (DA,  1998a,  1998b). 

The  products  cannot  simply  be  added  as  additional  training  requirements.  They  must 
replace  existing  but  less  efficient  or  effective  training.  One  commander  queried  “What  can  I  take 
off  my  plate  in  order  to  do  this?”  Organizations  like  the  Combat  Training  Centers  (CTCs)  must 
understand  the  purpose  of  the  FXXITP  and  its  products  so  they  can  provide  recommendations  to 
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units  on  which  FXXITP  product  they  can  train  with  to  overcome  problems  identified  during  LTP 
training  and  CTC  rotations. 

Thus,  despite  what  was  in  many  cases  a  good  understanding  of  the  products  and  acceptance 
that  the  FXXITP  products  could  help  them,  not  all  commanders  were  able  to  use  these  products 
in  lieu  of  other  training  events  required  by  FORSCOM  and  installation  training  regulations.  This 
seems  to  have  been  primarily  because  the  program  was  a  TRADOC  initiative  and  not  a  DA  and 
FORSCOM  supported  program. 

Lesson:  A  formal  program  of  education  and  orientation  should  be  implemented  to 

explain  the  FXXITP  products  to  the  installation  and  brigade  leadership,  future 
leaders,  and  training  institutions. 

Throughout  the  duration  of  the  project,  there  were  questions  about  the  purpose  of  the 
FXXITP  and  its  products.  Even  though  initial  orientation  briefings  and  product  T3  sessions  were 
conducted  at  both  user  unit  installations,  the  chain  of  command  at  both  installations  admitted  that 
they  never  really  understood  the  products  as  well  as  they  should  have.  Likewise,  the  NTC 
personnel  had  concerns  about  the  products  that  showed  an  incomplete  understanding  of  the 
product  intent,  the  product  purpose(s),  and  the  Army  expectations  regarding  the  use  and  support 
of  the  products  by  the  NTC  operations  group. 

One  common  problem  observed  throughout  the  project  dealt  with  a  unit’s  inability  to 
understand  and  implement  the  products  available  to  them  through  the  FXXITP.  The  initial 
concept  for  all  of  the  FXXITP  products  was  that  they  would  be  fully  exportable,  a  concept  that 
was  invalidated  during  the  trials  of  most  of  the  products  (C.  H.  Campbell,  Graves,  et  al.,  1998;  C. 
H.  Campbell  et  al.,  1999;  Graves  et  al.,  1997).  To  help  offset  this  the  IS  AT  team  conducted 
initial  orientation  briefings  for  the  installation  and  brigade  leadership.  These  orientation 
briefings  gave  these  leaders  a  basic  understanding  of  the  FXXITP  and  the  intent  for  each  product 
but  did  not  Walk  the  leadership  through  the  components  of  each  product.  In  addition  to  the  initial 
orientation  briefings,  the  ISAT  team  developed  and  conducted  a  T3  course  and  provided  the  unit 
with  a  supporting  guide  for  the  BSTS  and  vignette  products.  The  ISAT  team  also  assisted  the 
unit  during  the  preparation  and  train-up  phases  of  the  BBSE  exercise.  Yet,  as  stated  above, 
misunderstandings  continued. 

In  a  separate  effort,  the  ISAT  team  conducted  a  more  comprehensive  orientation  session 
with  representatives  of  7ATC.  Over  a  two-day  period,  the  ISAT  team  provided  a  hands-on 
orientation  of  the  BSTS,  vignettes,  BSE,  and  BBSE  products.  During  the  BSTS  and  vignette 
orientations,  the  ISAT  team  used  the  previously  developed  T3  course  and  supporting  guide  and 
then  developed  a  storyboard  for  walking  the  7ATC  representatives  through  each  component  of 
the  BSE/BBSE.  This  education  and  orientation  effort  was  felt  to  be  very  successful,  yet  it  was 
not  tied  to  a  specific  implementation  schedule.  Rather,  the  7ATC  representatives  requested  the 
orientation  for  their  own  information  and  evaluation.  Perhaps  the  key  to  success  is  in  the 
motivation  of  the  recipients — whether  they  are  participating  because  they  want  the  information, 
or  because  they  are  feeling  pressured  to  add  yet  more  briefings  to  an  already  overburdened 
schedule. 
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Lesson:  A  formal  program  of  maintenance  should  be  instituted  to  ensure  that  these 
products  remain  current  and  viable. 

One  of  the  biggest  complaints  by  the  participating  units  was  that  the  products  were  outdated. 
While  this  is  likely  to  occur  eventually  for  all  training  products  being  delivered  to  the  force,  the 
force  should  be  able  to  expect  that  this  problem  is  only  a  temporary  condition.  In  the  case  of  the 
FXXITP  products,  there  is  a  DTDD  effort,  currently  ongoing,  to  update  the  FXXITP  products. 

The  wider  lesson  applies  to  products  besides  those  evaluated  in  this  project.  All  structured 
training  development  (which  is,  by  definition,  training  that  is  tied  to  doctrinal  task  performance) 
should  have  a  built-in  plan  for  maintenance.  As  described  earlier,  maintenance  plans  for 
simulation-  or  computer-based  training  should  also  include  considerations  of  updating  the 
projects  in  synchronization  with  technological  advances. 

Lesson:  A  formal  program  of  sustainment  and  support  should  be  instituted  to  ensure 
that  units  are  able  to  adapt  the  products  as  needed  and  implement  them  with 
minimal  disruption  to  unit  ongoing  requirements. 

After  concerns  about  doctrinal  currency,  the  next  most  frequent  request  was  for  products 
that  could  be  tailored  to  unit  needs  and  situations.  A  plan  for  sustainment  of  the  products  could 
include  surge  teams  that  assist  first-time  users  or  help  with  modifications  to  products  to  meet  unit 
training  needs.  Lower  cost  off-site  assistance  could  be  offered  by  means  of  a  hot  line  or  help 
desk  and  a  web  page  for  disseminating  information. 

Lesson:  If  BSC  contractors  are  expected  to  support  FXXITP  product  implementation, 
then  their  contracted  logistics  support  ( CLS)  contracts  should  be  written  to 
include  that  requirement. 

Throughout  the  project,  there  was  a  great  deal  of  inconsistency  in  the  level  of  support 
available  at  different  simulation  centers.  One  center  said  that  they  didn’t  conduct  BBS  exercises; 
another  center  said  they  were  not  required  to  support  brigade-level  exercises  using  Janus.  This  is 
contrasted  to  the  experience  during  the  development  of  the  BSE/BBSE,  where  a  third  center  said 
they  could  support  anything  the  unit  put  in  front  of  them.  In  order  for  structured  training 
products  to  be  adequately  supported,  the  simulation  center  CLS  contracts  should  include 
language  requiring  support  of  the  structured  training  products  as  they  were  designed.  This  could 
also  be  expanded  to  include  support  requirements  for  Training  Support  Brigades  and  Divisions 
and  for  the  U.S.  Army  Reserve  Exercise  Brigades  that  could  provide  this  training  to  RC  and  AC 
units  in  the  future. 


Training  Effectiveness  Assessment 

On  two  important  objectives  of  the  project,  we  were  totally  unsuccessful.  We  were  unable 
to  see  even  one  complete  implementation  of  any  of  the  products,  and  we  were  unable  to  conduct 
a  rigorous  assessment  of  the  effectiveness  of  the  training  products.  As  described  in  Section  3 
and  earlier  in  this  section,  there  were  a  number  of  reasons:  competing  unit  demands  and 
missions,  misunderstandings  about  product  utility,  lateness  of  product  introduction  into  the  unit 
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training  calendars,  lack  of  understanding  of  command  emphasis.  Despite  the  presence  of 
training  coordinators  and  surge  teams,  despite  the  orientation  sessions  and  train  the  trainer 
sessions,  the  program  was  not  implemented. 

But  even  with  a  complete  implementation,  we  would  not  have  been  able  to  obtain  sufficient 
data  to  support  a  full  training  effectiveness  assessment  along  the  line  of  Kirkpatrick’s  model 
(1994).  One  or  two  brigades  with  ongoing  uncontrolled  activities,  continuous  personnel 
turnover,  and  the  absence  of  a  control  group,  would  not  represent  acceptable  conditions  for  such 
a  study.  Additionally,  many  of  the  data  sources  were  not  accessible:  NTC  observers  and  LTP 
personnel  protect  their  clientele  and  their  observations  closely,  quite  properly  not  permitting  any 
results  to  be  used  for  purposes  for  which  they  are  not  intended. 

This  is  not  to  say  that  the  assessment  work  yielded  no  useful  information.  A  qualitative 
assessment,  with  its  detailed  description  of  program  implementation,  can  be  the  method  of 
necessity  in  many  situations,  but  is  often  the  method  of  choice  (Patton,  1987).  Nonetheless,  we 
feel  keenly  the  need  for  a  description  of  impact. 

Lesson:  We  are  still  in  need  of  a  rigorous  training  effectiveness  assessment  of  the 
FXXITP  products. 

Perhaps  the  Kirkpatrick  model  (1994)  is  not  the  correct  one  for  this  kind  of  training 
evaluation.  Quasi-experimental  methods,  such  as  are  propounded  in  D.  T.  Campbell  and 
Stanley’s  classic  work  (1966)  and  updated  in  Cook  and  Campbell  (1979),  should  be  explored  as 
more  appropriate  possibilities  for  assessment  studies.  We  ought  not  to  abandon  the  attempt.  The 
guidance  in  a  variety  of  guides  to  program  evaluation  (e.g..  Joint  Committee  on  Standards  for 
Educational  Evaluation,  1994;  King  et  al.,  1987;  Patton,  1987;  Rossi  &  Freeman,  1993;  Shadish 
et  al.,  1991)  should  be  explored  carefully  to  determine  whether  there  are  other  approaches  that 
could  be  incorporated.  As  we  continue  to  develop  structured  training  products,  we  should  also 
continue  the  attempt  to  obtain  evidence  of  their  effectiveness. 

Summary 

This  section  has  presented  discussions  of  the  data  summarized  in  Section  4,  and  proposed  a 
number  of  lessons  learned.  The  lessons  address  considerations  of  acceptability,  impact, 
supportability,  and  training  effectiveness  assessment  itself.  The  lessons  are,  on  one  level, 
discouraging,  as  they  are  generally  admissions  of  shortcomings  in  the  FXXITP  products.  Within 
the  lessons,  however,  we  have  attempted  to  offer  solutions  to  the  identified  problems.  Some  of 
the  solutions  are  more  ambitious  than  others:  Some  require  additional  research,  while  others  will 
require  action  at  the  highest  levels  of  Army  leadership.  No  one  solution  can  solve  all  of  the 
problems,  and  no  one  solution  can  be  truly  effective  for  even  one  problem  without 
implementation  of  solutions  that  address  all  of  the  problems. 

Those  solutions  that  are  within  the  reach  of  training  designers  and  developers  include 
research  on  TSP  and  implementation  models,  redesign  of  TSP  products  and  distribution 
requirements,  and  planning  for  maintenance  and  sustainment  of  products.  Other  solutions, 
including  the  need  for  command  emphasis  at  division-level  and  higher,  and  the  institutionalizing 
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of  the  program,  will  require  that  TRADOC  and  Army  leaders  make  a  commitment  to  support  the 
development  and  implementation  of  such  products  that  may  increase  readiness  without 
increasing  training  costs. 


Section  6.  Conclusions  and  Recommendations 

In  Section  5,  we  presented  a  set  of  13  lessons,  derived  from  observations,  interviews,  and 
analysis  of  the  questionnaire  data.  The  lessons  can  be  further  reduced  to  three  critical  issues  that 
demand  attention,  based  on  this  research  effort: 

•  plans  for  fielding,  maintenance,  and  sustainment; 

•  methods  for  making  TSPs  more  usable;  and 

•  Army  and  TRADOC  initiatives  for  ensuring  product  use. 

Plans  for  Fielding,  Maintenance,  and  Sustainment 

Appendix  D  contains  a  briefing  prepared  for  DTDD  describing  the  initial  fielding, 
maintenance,  and  sustainment  planning,  and  Appendix  E  outlines  the  plan  with  more  detail;  this 
plan  forms  the  basis  of  the  ongoing  effort  within  DTDD.  Figure  3  illustrates  the  relationships 
among  the  processes.  As  shown,  initial  fielding  gradually  transitions  to  ongoing  sustainment. 
Similarly,  initial  updates  are  succeeded  by  a  process  of  continuing  updates.  Feedback  and 
lessons  learned  during  fielding  are  used  to  make  the  initial  updates,  which  are  then  delivered  and 
incorporated  into  the  already  fielded  products.  This  process  continues  over  time:  Units  use  the 
products  and  provide  suggestions  or  concerns,  and  developers  use  the  information  to  make 
continual  improvements.  Although  solutions  for  each  part  of  the  process  can  be  planned  and 
executed  separately,  all  three  needs  must  be  addressed  in  order  for  any  solution  to  be  effective. 


Figure  3.  Integration  of  fielding,  maintenance,  and  sustainment  activities  for  Force  XXI 
Training  Program  products. 
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The  plan  calls  for  a  combination  of  hot  lines  and  surge  teams  to  work  with  first  users.  The 
same  mechanisms  would  also  be  useful  for  obtaining  feedback  from  units  using  the  materials,  for 
use  in  planning  for  and  carrying  out  the  most  urgently  needed  updates.  As  the  fielding  and 
implementation  support  are  continuing,  the  update  team  would  be  amassing  observation, 
feedback,  and  lessons  learned  concerning  the  products  and  their  use.  This  information  would  be 
used  to  determine  the  needs  for  additional  maintenance  to  the  products. 

The  requirements  demand  an  integrated,  multifaceted  plan  that  addresses  initial  fielding 
support,  immediate  updates,  continuing  maintenance,  and  ongoing  sustainment  for  users.  The 
plan  must  take  into  consideration  primary  users  in  the  AC  and  RC  units,  as  well  as  institutional 
users  and  supporters.  Finally,  the  plan  must  ensure  that  the  personnel  who  will  carry  out  the 
activities  are  of  the  appropriate  levels  of  expertise  and  experience  with  respect  to  doctrine, 
technology,  and  the  training  products  themselves. 

Making  Training  Products  More  Usable 

The  objective  is  worthy:  TSP  materials  should  be  extremely  user  friendly  and  should 
present  no  barriers  to  use.  For  years,  training  researchers  have  been  working  on  visions  of 
materials  that  participants  would  use  as  designed.  Despite  design  innovations  and  increasingly 
simplified  and  comprehensive  presentations,  however,  we  find  repeatedly  that  guides  and 
instructions  are  not  used.  Directed  research  on  what  works,  for  what  products,  under  what 
conditions,  would  be  useful  in  addressing  this  objective.  But  the  FXXITP  and  the  Army  should 
also  be  prepared  to  insist  that  users  attempt  to  use  the  products.  Until  we  get  a  committed  effort 
at  implementation,  we  are  still  guessing  at  optimal  formats  and  models. 

Army  Emphasis  on  Product  Incorporation  in  Training  Plans 

All  we  can  do  with  this  recommendation  is  to  ask  decision-makers  to  make  their  decision: 

Do  they  want  the  products  to  be  used?  If  so,  then  they  must  provide  the  resources — time  and 
personnel— and  the  edict  that  tells  units  that  the  products  are  to  be  used.  They  must  invest  in  a 
fielding,  maintenance,  and  sustainment  plan  such  as  the  one  in  Appendix  E  that  will  keep  the 
products  viable. 

As  institutionalized  training  programs,  the  LTP,  CTC  rotations.  Battle  Command  Battle 
Staff  Training  for  RC  units,  and  other  such  events  “work.”  Why  is  that?  These  training 
programs  have  an  established  infrastructure  of  expert  personnel  and  tested  training  approaches 
that  ensures  support  for  user  units.  They  are  a  part  of  the  system:  Units  know  that  they  must 
participate,  and  are  eager  to  do  so.  The  programs’  reputation  for  effective  rotation  logistics  and 
tough,  thorough  training  ensure  units  that  their  time  will  be  well  spent.  Since  the  CTCs  and 
other  similar  programs  began  to  be  a  part  of  the  Army  training  system  (in  the  broadest  sense), 
they  have  become  ingrained  in  the  training  culture. 

This  should  be  the  vision  for  the  FXXITP.  It  will  take  time  and  foresight  and  commitment 
of  resources  and  planning.  Insights  drawn  from  this  project  should  be  useful  in  thinking  through 
the  infrastructure  and  education  needs.  Lessons  learned  from  the  earliest  days  of  CTC  history 
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would  also  inform  the  planning  process.  Finally,  research  from  organizations  such  as  ARI  could 
continue  to  guide  the  institutionalization  of  the  program. 

Summary 

The  ISAT  project  has  provided  a  wealth  of  valuable  information  to  the  FXXITP  and  the 
larger  training  community.  The  quantitative  data  were  relatively  sparse,  compared  to  the 
richness  of  the  observations  and  interviews.  But  in  training  effectiveness  assessments,  even 
when  greater  amounts  of  quantitative  data  are  available,  the  analyses  and  results  must  be 
supplemented  with  observations  and  comments.  Thus  any  disappointment  with  the  amount  of 
data  is  more  than  relieved  by  satisfaction  with  the  qualitative  information,  the  honest  and 
constructive  input  from  users,  and  the  insights  provided  by  reviewers.  The  recommendations 
and  lessons  learned  should  assist  the  FXXITP,  TRADOC,  and  the  Army  to  continue  their 
attempts  to  provide  training  that  improves  readiness  and  is  accessible  by  units. 
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A2C2 

AAR 

AC 

ADA 

AFRU 

ARI 

ASAS 

ASAT 

ATC 

AUSA 

BBS 

BBSE 

BCTP 

BLUFOR 

BSC 

BSE 

BSTS 

CAC 

CALL 

CATS 

CBI 

CCTT 

CLS 

COA 

COBRAS 

COL 

COMPS 

COR 

COTS 

CP 

CSS 

CTC 

DA 

DCST 

DISCOM 

DMA 

DTDD 


Appendix  A 
Acronym  List 

Army  Airspace  Command  and  Control 

after  action  review 

active  component 

air  defense  artillery 

Armored  Forces  Research  Unit 

U.S.  Army  Research  Institute  for  the  Behavioral  and  Social  Sciences 

All  Source  Analysis  System 

Automated  Systems  Approach  to  Training 

Army  Training  Center 

Association  of  the  United  States  Army 

Brigade/Battalion  Battle  Simulation 

Brigade  and  Battalion  Staff  Exercise 

Battle  Command  Training  Program 

Blue  Forces 

battle  simulation  center 

Brigade  Staff  Exercise 

Battle  Staff  Training  System 

Combined  Arms  Center 
Center  for  Army  Lessons  Learned 
Combined  Arms  Training  Strategies 
computer-based  instruction 
Close  Combat  Tactical  Trainer 
contracted  logistics  support 
Course  of  Action 

Combined  Arms  Operations  at  Brigade  Level,  Realistically  Achieved  Through 
Simulation 
colonel 

comprehensive  assessment  component 
contracting  officer’s  representative 
commercial  off  the  shelf 
command  post 
combat  service  support 
Combat  Training  Center 

Department  of  the  Army 
Deputy  Chief  of  Staff  for  Training 
Division  Support  Command 
Defense  Mapping  Agency 

Directorate  of  Training  and  Doctrine  Development 
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EMMii 

EXCON 


Environment  for  MultiMedia  interactive  instruction 
exercise  control 


FORSCOM  Forces  Command 

FRAGO  fragmentary  order 

FSO  fire  support  officer 

FTX  field  training  exercise 

FXXITP  Force  XXI  Training  Program 

HumRRO  Human  Resources  Research  Organization 


IS  AT  Implementation  and  Support  for  the  Assessment  of  Force  XXI  Training 

Program 

ISD  Instructional  Systems  Design 

LAN  local  area  network 

LTC  lieutenant  colonel 

LTP  Leader  Training  Program 

NCO  noncommissioned  officer 

NTC  National  Training  Center 


O/C  observer/controller 

OPFOR  opposing  forces 

OPORD  operation  order 


PCC  pre-command  course 

PO  performance  objective 

QC  quality  control 


R&S 

RC 

RSOI 

RTB 


reconnaissance  and  surveillance 
reserve  component 

reception,  staging,  onward  movement,  and  integration 
Regional  Training  Brigade 


51 

52 

53 

54 
SAT 
SGE 
SIMEX 
SIMNET 
SME 
SOP 


personnel  officer 
intelligence  officer 
operations  and  training  officer 
supply/logistics  officer 
Systems  Approach  to  Training 
Staff  Group  Exercises 
simulation  exercise 
Simulation  Networking 
subject  matter  expert 
standing  operating  procedure 
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SPSS 

Statistical  Package  for  the  Social  Sciences 

T3 

TMS 

TOC 

TOE 

TRADOC 

TSB 

TSC 

TSP 

train-the-trainer 

training  management  system 

tactical  operations  center 

Table  of  Organization  and  Equipment 

U.S.  Army  Training  and  Doctrine  Command 
Training  Support  Brigade 
training  support  coordinator 
training  support  package 

USAARMC 

USAREUR 

U.S.  Army  Armor  Center 

U.S.  Army  Europe 

WARNO 

warning  order 

Y2K 

year  2000 

XO 

executive  officer 
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Appendix  B 

Assessment  Questions  of  Interest 


Level  I  —  Reaction:  Acceptance  by  the  Training  Audience 

=>  Credibility 

To  what  extent  are  the  training  materials  consistent  with  current  doctrine? 

To  what  extent  would  battle  staff  officers  recommend  the  training  support  packages 
(TSPs)  to  their  counterparts  in  other  units? 

=>  Ease  of  Use 

How  easy  is  it  to  use  are  the  various  TSP  materials? 

How  acceptable  is  the  time  required  to  prepare  and  execute  the  training? 

How  acceptable  are  the  unit  support  requirements  (personnel,  equipment,  facilities)? 

=>  Form  and  Function 

How  useful  are  the  job  aids  provided  in  the  TSPs?  What  others  are  needed? 

What  is  the  quality  and  timeliness  of  performance  feedback? 

How  acceptable  is  the  degree  of  control  exercised  by  trainers/training  audience? 

How  acceptable  is  the  training  flexibility  provided  by  the  TSPs?  [Examine  scheduling, 
mission/task  selection,  entry  points,  sequencing,  small  group  composition,  support 
requirements,  etc.] 

How  acceptable  are  the  media  used  for  the  various  components  of  the  TSPs? 

How  acceptable  is  the  hardware/software  (including  reliability  and  user  friendliness)? 

=>  Product  Value 

What  are  the  strengths  of  the  TSPs?  The  weaknesses? 

How  do  users  feel  about  the  quality  of  the  various  TSPs  and  their  components? 

How  likely  are  commanders  and  training  officers  to  use  the  TSPs? 

=>  Potential  Enhancements 

What  problems  are  encountered  in  using  the  TSPs? 

What  enhancements  would  increase  ease  of  use,  flexibility,  trainer  control,  user  control, 
effectiveness,  and  feedback? 

Level  2  —  Learning:  Achievement  of  Training  Objectives 

=»  Suitability  of  Training  Objectives 

What  knowledge/skills/tasks  (individual,  collective)  do  the  TSPs  train  well? 

How  clearly  stated  are  the  training  objectives? 

Do  the  TSPs  focus  on  the  correct  knowledge/skills/tasks  for  a  given  staff  officer/section? 
What  critical  knowledge/skills/tasks  should  be  added?  Which  should  be  eliminated? 

To  what  extent  do  skills/abilities  transfer  across  the  various  training  products  being 
evaluated  (i.e.,  mutual  linkage/support  among  the  four  products)? 

=>  Training  Effectiveness 

How  well  do  the  TSPs  establish  the  conditions  for  achieving  the  training  objectives? 
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Does  the  performance  feedback  focus  on  the  correct  topics? 

How  challenging  is  the  training?  How  appropriate  is  the  pace? 

What  improvements  or  sustainments  in  knowledge/skills/abilities  occur? 

What  level  of  mastery  is  demonstrated  during  individual  (Battle  Staff  Training  System) 
training? 

How  does  TSP-driven  training  impact  the  individual’s  understanding  of  his/her  position? 
Of  his/her  staff  section?  Of  the  entire  staff? 

How  much  does  training  with  the  TSPs  improve  the  individual’s  confidence  in  his/her 
critical  skills  and  abilities?  In  the  abilities  of  others  with  whom  they  work? 

How  effective  are  the  train-the-trainer  materials,  including  training  management 
information? 

=>  Potential  Enhancements 

How  can  the  training  objectives  or  their  presentation  be  improved? 

How  can  the  TSPs  be  improved  to  better  support  the  training  objectives? 

What  other  vignettes  should  be  developed  to  better  link  the  vignettes  to  Brigade  and 
Battalion  Staff  Exercise  (BBSE)  performance  objectives? 

What  changes  are  needed  to  meet  future  training  objectives? 

How  should  the  training  objectives  be  expanded  to  support  training  of  digital  tasks? 

Level  3  —  Behavior:  Impact  on  Job  Performance 

=>  Job  Relevance 

To  what  extent  are  TSP-based  knowledge/skills/abilities  applicable  to  job  performance? 
To  what  extent  do  the  TSPs  enable  practice  of  skills  that  would  be  used  during  battle? 
How  useful/valuable  are  the  knowledge/skills/tasks  trained? 

=>  Impact  on  Staff  Competence 

What  aspects  of  job  performance  are  impacted  by  training  with  the  TSPs? 

How  does  training  with  the  TSPs  impact  teamwork  and  effectiveness  of  staff  sections? 

How  much  does  training  with  the  TSPs  enhance  the  confidence  of  commanders  and 
primary  staff  officers  in  the  proficiency  of  their  staff(s)? 

=>  Impact  on  Staff  Performance 

To  what  extent  is  the  individual/section/unit  prepared  for  a  Combat  Training  Center-like 
experience? 

What  skills,  tasks,  section/group  activities,  etc.  appear  to  be  enhanced  as  the  National 
Training  Center  (NTC)  rotation  starts?  Which  of  these  enhancements  result  from  TSP- 
driven  training? 

To  what  extent  does  TSP-driven  training  enhance  staff  proficiency  and  effectiveness  at 
the  start  of  NTC  training? 

What  level  of  staff  process  performance  is  evident  during  the  first  NTC  mission?  During 
the  last? 

What  battle  outcomes  are  evident  during  the  various  NTC  missions? 
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What  performance  strengths  and  weaknesses  are  noted  in  NTC  (or  Brigade  Staff  Exercise 
[BSE]/BBSE)  after  action  reviews? 

To  what  extent  do  knowledge/skills/abilities  acquired  in  using  the  TSPs  enhance  job 
performance  at  the  NTC  (or  in  command  post  exercises)? 

How  does  TSP-driven  training  contribute  to  unit  performance  at  the  NTC? 

=>  Potential  Enhancements 

How  could  the  TSPs  be  improved  to  impact  job  performance  more  favorably? 

How  should  the  TSPs  be  modified  to  meet  job  performance  requirements  of  the  future? 
What  enhancements  are  needed  to  meet  job  performance  requirements  of  digital  units? 

Level  4  —  Results:  Impact  on  Organizational  Objectives  and  Resources 

=$  Relevance  to  Unit  Goals  and  Requirements 

How  do  the  TSPs  facilitate/hinder  preparations  for  an  NTC  rotation? 

How  well  do  the  training  outcomes  of  the  various  products  compare  with  the  unit  training 
goals? 

How  does  training  with  the  TSPs  impact  unit  readiness? 

Where  does  each  of  the  products  fit  into  the  unit  training  strategy? 

To  what  extent  are  the  TSPs  compatible  with  existing  Combined  Arms  Training 
Strategies  (CATS)? 

How  should  the  TSPs  be  incorporated  into  long-range  training  plans?  Into  CATS? 

=>  Cost-Benefit  Considerations 

What  does  it  cost  (time,  funds,  support  requirements,  etc.)  to  use  the  training  products 
(preparation  and  execution)? 

How  do  the  TSPs  compare  (in  terms  of  time,  fiscal  cost,  support  requirements,  ease  of 
use,  fidelity,  value  added,  etc.)  with  other  training  tools  available  for  similar  purposes? 

To  what  extent  do  the  training  products  save  the  unit  time,  funds,  etc.  in  preparing  for  an 
NTC  rotation? 

What  is  the  likely  resource  impact  of  using  the  TSPs?  What  resource  problems  are 
anticipated? 

To  what  extent  do  the  benefits  of  the  training  outweigh  the  costs? 

How  valuable  would  it  be  for  the  Army  to  field  the  TSPs? 

=>  Potential  Enhancements 

How  could  the  TSPs  be  modified/expanded  to  enhance  their  value  to  units? 

What  steps  should  be  pursued  to  meet  future  training  needs? 

How  should  the  products  be  enhanced  to  support  training  for  digital  operations? 

Implementation:  Utilization,  Implementation  and  Support  Issues 

=>  Training  Management  and  Product  Utilization 

How  do  units  manage  selection,  scheduling,  and  preparation  for  TSP-driven  training? 

With  what  frequency,  timeliness  and  sequencing  do  individuals/units  use  the  TSPs? 

How  much  time  is  taken  to  prepare  and  complete  the  training?  Is  it  realistic? 
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How  perishable  are  the  knowledge  and  skills  trained  in  the  TSPs? 

How  often  should  sustainment  (refresher)  training  occur  with  the  TSPs? 

=>  Compliance  with  TSP  Procedures 

To  what  extent  do  individuals/units  comply  with  TSP  instmctions  (preparation  and 
execution)? 

What  personnel  serve  as  observer/controllers  for  training  with  products? 

What  personnel  staff  the  Division  Response  Cell  during  BSE  and  BBSE  exercises? 
How  do  Simulation  Center  personnel  train/prepare  for  TSP  exercises? 

What  training  environment  factors  (e.g.,  turnover,  staffing  difficulties,  time  constraints, 
command  emphasis)  influence  the  use  of  the  TSPs? 

What  misunderstandings  do  the  users  have  regarding  the  training  products? 

=>  Support  Requirements 

What  kinds  of  support  do  individuals/units  request  from  the  IS  AT  Team?  From 
Simulation  Center  personnel? 

What  is  the  quality  of  implementation  support  provided  by  the  ISAT  Team?  By 
Simulation  Center  personnel? 

What  are  the  minimum  essential  support  requirements  for  fielding? 

What  steps  can  be  taken  to  facilitate  implementation  when  the  ISAT  Team  is  no  longer 
available? 

=>  Implementation  Considerations 

Where  should  training  with  the  various  TSPs  take  place  (schoolhouse,  unit,  etc.)? 

What  problems  are  encountered  in  implementing  the  TSPs  (distribution,  equipment, 
facilities,  management,  exercise  control,  etc.)? 

How  exportable  are  the  hardware/software  components? 

What  hardware/software  modifications  are  needed  to  support  fielding? 

What  obstacles  stand  in  the  way  of  implementing  the  TSPs? 

=>  Lessons  Learned 

What  innovative  practices  and  techniques  do  users/trainers  develop? 

What  innovative  uses  or  applications  do  units  find  for  the  training  products? 

What  lessons  learned  will  benefit  future  development  and  implementation  efforts? 
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Appendix  C 

Sample  Data  Collection  Instruments 


Document  Title 

Page 

ISAT  Survey  -  Training  Audience  Feedback:  Battle  Staff  Training  System 

C-2 

ISAT  Survey  -  Training  Audience  Feedback:  Post-National  Training  Center 

C-8 

ISAT  Structured  Interview  Guide:  BBSE  Execution 

C-17 

ISAT  Observer’s  Guide:  BBSE  Execution 

C-24 

C-l 


ISAT  SURVEY 


Training  Audience  Feedback:  Battle  Staff  Training  System 

PT  No.  60-1 2a 


Instructions 

This  survey  asks  for  your  opinions  about  the  BSTS  course  you  just  completed.  The 
survey  gives  you  the  opportunity  to  influence  this  Force  XXI  Training  Program  product 
through  your  comments  and  suggestions. 

Please  answer  each  question  based  on  your  experiences  as  you  worked  through  the 
course. 

Information  obtained  from  this  survey  will  be  used  for  training  effectiveness  analysis 
purposes.  Neither  you  nor  your  unit  will  be  referenced,  without  permission,  in  any 
briefing  or  publication. 

Thank  you  for  your  input. _ 


DATA  REQUIRED  BY  THE  PRIVACY  ACT  OF  1 974 
AUTHORITY:  Title  10,  USC,  Sec  4503 

PRINCIPAL  PURPOSE:  The  data  collected  are  to  be  used  for  assessment  purposes 
only. 

ROUTINE  USE:  The  data  collected  under  the  ISAT  contract  effort  are  to  be  used  by 
the  U.S.  Army  Research  Institute  for  the  Behavioral  and  Social  Sciences  pursuant  to  its 
research  mission  as  prescribed  in  AR  70-1 .  When  identifiers  (name  or  personal 
identification  number)  are  requested,  they  are  for  administrative  and  data  control 
purposes  only.  Full  confidentiality  of  the  responses  will  be  maintained  in  processing 
these  data. 

DISCLOSURE:  You  will  be  asked  to  provide  feedback  on  your  use  of  the  FXXITP 
training  support  packages  and  how  well  they  helped  you  accomplish  unit  training. 

Your  participation  in  this  assessment  is  strictly  voluntary.  You  are  encouraged  to 
provide  complete  and  accurate  information  in  the  interest  of  the  assessment,  but  there 
will  be  no  effect  on  you  for  not  providing  any  part  of  the  information. 

Please  sign  below  to  indicate  that  you  agree  to  participate  in  this  data  collection  effort. 

Print  Name:  _ _ _  Date:  _ 

Signature: 


Date: 


^Please  enter  the  first  letter  of  your  last  name  plus  the  last  four  digits  of  your  SSN: 


1 .  What  BSTS  course  did  you  take?  _ 

2.  What  is  your  rank?  _ 

3.  What  is  your  branch  specialty  or  MOS?  _ 

4.  What  is  your  current  duty  position?  _ 

5.  How  long  have  you  been  in  your  current  duty  position? 


6.  Which  military  courses  have  you  completed?  (Circle  all  that  apply) 
OBC  OAC  CAS3  C&GSOC  SAMS 
Other  (please  specify):  _ 


AWC 


7.  Which  Officer  Advanced  Course  did  vou  complete?  Circle  one. 


Armor  I  Infantry  Field  Arty  ADA  V  Signal 


Mil  Intel  Quartermaster  Transp’n  Chemical 


Other  (please  specif 


Aviation 


8.  Please  list  your  previous  Bn/Bde/Div  staff  assignments  (e.g.,  Bn  S4 


9.  How  many  times  have  you  participated  in 
the  following  training  events? 


a.  National  Training  Ctr  (NTC)  rotation 


b.  Joint  Readiness  Training  Ctr  (JRTC)  rotation 


c.  Cbt  Maneuver  Training  Ctr  (CMTC)  rotation 


d.  Leader  Training  Program  (LTP 


e.  Warfighter 


f.  Battle  Command  Training  Program  (BCTP 


.  Bde  Command  &  Battle  Staff  Training  (BCBST 


h.  Janus  and  BBS  staff  exercises 


Circle  one  for  each  item: 

Never 

Once 

Twice 

Three  >  Tf- 

Times  Tirr 

0 

i 

2 

3  ^ 

0 

i 

2 

3  A 

0 

i 

2 

3  A 

0 

i 

2 

3  A 

0 

i  i 

2 

3  A 

0 

i 

2 

3  A 

0 

i  i 

2 

3  4 

0 

!  i 

2 

3  A 

Acceptability  of  the  Training  Materials 


Circle  one  for  each  item: 


1 0.  How  easy/difficult  was  it  to  use 
the  following? 


Did  not  | 
Use 


Difficult 
to  Use 


Neutral 


a.  Student  Guide 

0  1 

1 

2 

3 

b.  Job  Aids 

o  ! 

1 

2 

3 

c.  Computer-Based  Modules 

o  I 

1 

2 

3 

d.  FMs,  other  References 

o  1 

1 

2 

3 

Very  Easy 
to  Use 


Did  not 
Use 


Circle  one  for  each  item: 


Totally 

Inconsistent  Neither 


1  la.  To  what  extent  were  the  following 
consistent  with  current  doctrine? 


Course  Contents  I  0 


2)  Division/Briqade  OPORD 


3)  Graphic  Overlays 


4)  Practical  Exercises 


11b.  What  changes  would  make  these  materials  more  consistent  with  doctrine? 


Totally 

Consistent 


12.  Our  BCT  /  BN  should  use  the  BSTS 

Don’t 

Strongly 

Strongly 

courses  for  training  in  the  future. 

Know 

Disagree  ^ 

Neutral  ^  Agree 

Circle  one:  1 

0 

1 
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13. 1  recommend  that  other  BCTs  or  BNs  use 

Don’t 

Strongly 

Strongly 

the  BSTS  courses  for  their  training. 

Know 

Disagree  ^ 

Neutral  ^  Agree 

Circle  one: 

0 

1 
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Achievement  of  Training  Objectives 


1 4.  To  what  extent  were  your  course’s 

Circle  one  for  each  item: 

training  objectives: 

To  N  o 

1  Extent 

Slight 

Extent 

Moderate 

Extent 

Great 

Extent 

To  a  Very 
Great  Extent 

a.  Clearly  stated? 

i  i 

2 

3 

4 

5 

b.  Focused  on  the  right  tasks  for  my  position? 

1  i 

2 

3 

4 

5 

c.  Appropriate  for  self-study? 

U 

2 

3 

4 

5 

15.  Please  list  topics  or  materials  that  should  be  added  to  the  course: 


1 6.  The  course  materials,  practical  exercises, 
exams,  etc.  established  the  conditions  for 
successful  training. 

Circle  one: 

To  No  Slight  Moderate  Great  To  a  Very 

Extent  Extent  Extent  Extent  Great  Extent 

1  2  3  4  5 

« - - - — - — - — _ 

17.  To  what  extent  did  the  course: 

Circle  one  for  each  item: 

To  No  Slight  Moderate  Great  To  a  Very 

Extent  Extent  Extent  Extent  Great  Extent 

a.  Improve  my  understanding  of  mv  tasks? 

12  3* 

5 

b.  Improve  mv  ability  to  perform  mv  tasks? 

12  3* 

5 

c.  Improve  my  understanding  of  the  staff? 

12  3* 

5 

d.  Provide  effective  feedback? 

12  3* 

5 

18.  Please  list  the  areas  in  which  you  gained  substantial  knowledge: 


19.  How  wo 


Impact  on  Job  Performance 


20.  Indicate  the  extent  to  which  this 
course: _ 


a.  Gave  me  knowledge/skills  that  are  appli¬ 
cable  to  mv  job  performance  in  combat. 


b.  Provided  information  that  I  value  hiahlv. 


c.  Enabled  me  to  practice  techniques  that  I 
would  use  durina  battle. 


|  To  No  Slight 
|  Extent  Extent 


Circle  one  for  each  item. _ 

Slight  Moderate  Great  To  a  Very 

Extent  Extent  Extent  Great  Extent 


1 

2 

3 

1 

2 

3 

1 

2 

3 

Impact  on  Unit  Training  Program 


22.  To  what  extent  does  this  course: 


a.  Help  our  BCT  or  BN  meet  its  staff  training 
goals? 


b.  Make  me  better  prepared  for  an 
LTP/NTC  rotation? 


c.  Save  training  time  for  our  BCT  or  BN? 


d.  Save  training  funds  for  our  BCT  or  BN? 


e.  Enhance  our  BCT’s  combat  readiness? 


Circle  one  for  each  item: 


To  No  Slight  Moderate  Great  To  a  Very 

Extent  Extent  Extent  Extent  Great  Extent 


1  2  3  4  5 


23.  Please  list  the  two  most  i 


the  staff  receives  from  BSTS  training : 


24a.  To  what  extent  do  the  benefits  of  this 

To  No 

Slight 

Moderate 

Great 

To  a  Very 

staff  training  course  outweigh  the  costs? 

Extent 

Extent 

Extent 

Extent 

Great  Extent 

Circle  one: 

1 

2 

3 

4 

5 

24b.  If  you  circled  “1”  or  “ 


25.  Given  the  training  value  of  this  course, 

Totally 

Totally 

the  amount  of  time  spent  is: 

Unacceptable  £ 

Neutral  ^  Acceptable 

(Circle  one) 

i 
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General  Comments 


28. . Please  provide  additional  suggestions  and  comments  about  the  course. 


IS  AT  SURVEY 


Training  Audience  Feedback:  Post-National  Training  Center 

PT  No.  60-1 2v 


INSTRUCTIONS 

This  survey  asks  for  your  opinions  about  the  Force  XXI  training  products  and  the  way 
they  impacted  your  NTC  rotation.  The  survey  gives  you  the  opportunity  to  influence  the 
training  products  through  your  comments  and  suggestions. 

Please  answer  each  question  based  on  your  experience  with  the  BSTS  courses, 
COBRAS  Vignettes,  Brigade  Operations  Exercise  (Oct  98  in  Janus),  and  Brigade/ 
Battalion  Staff  Exercise  (Jan  99  in  BBS)  as  they  relate  to  your  NTC  experience. 
Information  obtained  from  this  survey  will  be  used  for  training  effectiveness  analysis 
purposes.  Neither  you  nor  your  unit  will  be  referenced,  without  permission,  in  any 
briefing  or  publication. 

Thank  you  for  your  input. 


DISCLOSURE  REQUIRED  BY  THE  PRIVACY  ACT  OF  1 974 
AUTHORITY:  Title  10,  USC,  Sec  2358 

PRINCIPAL  PURPOSE:  The  data  collected  are  to  be  used  for  assessment  purposes 
only. 

ROUTINE  USE:  The  data  collected  under  the  ISAT  contract  effort  are  to  be  used  by 
the  U.S.  Army  Research  Institute  for  the  Behavioral  and  Social  Sciences  pursuant  to  its 
research  mission  as  prescribed  in  AR  70-1 .  When  identifiers  (name  or  personal 
identification  number)  are  requested,  they  are  for  administrative  and  data  control 
purposes  only.  Full  confidentiality  of  the  responses  will  be  maintained  in  processing 
these  data. 

DISCLOSURE:  You  will  be  asked  to  provide  feedback  on  your  use  of  the  Force  XXI 
training  products  and  how  well  they  helped  you  accomplish  unit  training. 

Your  participation  in  this  assessment  is  strictly  voluntary.  You  are  encouraged  to 
provide  complete  and  accurate  information  in  the  interest  of  the  assessment,  but  there 
will  be  no  effect  on  you  for  not  providing  any  part  of  the  information. 
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Date: 


^  Please  enter  the  1  st  letter  of  your  last  name  plus  the  last  4  digits  of  your  SSN: _ _ 

1 .  What  is  your  rank?  _ 

2.  What  is  your  branch  specialty  or  MOS?  _ 

3.  What  is  your  unit  of  assignment?  (circle  one)  Bde  HQ  1/34  AR  2/34  AR 

1/5  FA  1  EN  101  FSB  Other _ 

4.  What  is  your  current  duty  position?  _ 

5.  How  long  have  you  been  in  your  current  duty  position?  _ yrs  / _ mos 

6.  How  long  have  you  been  assigned  to  the  1  BCT/1  ID?  _ yrs  / _ mos 

7.  How  much  time  have  you  spent  as  a  Div/Bde/Bn  staff  officer?  _ _ yrs  / _ mos 


8.  Which  Battle  Staff  1 

[raining  System  (BSTS)  courses  did  you  take?  (Circi 

e  all  that  apply) 

Cdr 

XO 

S2 

S3 

S4 

S5 

FSO 

ENG 

ADAO 

SIGO 

CHEMO 

S3-Air 

Common  Core 

9.  In  which  Vignettes  did  you  participate  previously?  (Circle  all  that  apply) 
Mission  Analysis  Develop  Concept  of  Svc  Spt 

_ COA  Analysis _ Other 


1 0.  What  role  did  you  play  in  the  Bde  Ops  Exercise  (Oct  98  -  Janus)?  (Circi 

e  one) 

Main  CP  Staff 

Rear  CP  Staff  2/34  TOC  1 01  FSB  TOC  1  /5  FA  TOC 

1  EN  TOC 

Workstation 

Roleplayer 

Other 

Did  Not 
Participate 

1 1 .  What  role  did  you  play  in  the  Bde/Bn  Staff  Exercise  (Jan  99  -  BBS)?  (Circle  one) 

Bde  TOC 

1  /34  TOC  2/34  TOC  1 01  FSB  TOC  1/5  FA  TOC 

1  EN  TOC 

Other 

Did  Not 
Participate 

12a.  Did  you  fill  your  normal  duty  position  during  the  NTC  rotation? 

_ Yes _ No _ 

12b.  If  you  circled  “No”,  please  specify  the  role  you  filled: 
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Preparation  for  the  NTC 


***  In  questions  13, 14,  and 

1 15  please  use  the  following 

1  scale: 

Don’t  Know 

Very  Weak 

Weak 

Neither 

Strong 

Very  Strong 

0 

1 

2 

3 

4 

5 

13.  Rate  the  staff  processes  of  your  element  in  June  98  (Cdr/XO  rate  entire  staff,  S3  rate  S3 

Staff  section,  FSO  rate  FSE,  etc.):  (Circle  one  value  in  every  cell  under  the  plan-monitor-direct  columns) 

PLAN 

MONITOR 

DIRECT 

a.  Reconnaissance  operations 

0  1  2  3  4  5 

0  1  2  3  4  5 

0  1  2  3  4  5 

b.  Parallel  planning 

0  1  2  3  4  5 

0  1  2  3  4  5 

0  1  2  3  4  5 

c.  Information  management 

0  1  2  3  4  5 

0  1  2  3  4  5 

0  1  2  3  4  5 

d.  Integration  of  fires 

0  1  2  3  4  5 

0  1  2  3  4  5 

0  1  2  3  4  5 

e.  Accelerated  decision  making 

0  1  2  3  4  5 

0  1  2  3  4  5 

0  1  2  3  4  5 

14.  Rate  the  staff  processes  of  your  element  in  October  98  (Cdr/XO  rate  entire  staff,  S3  rate 

S3  Staff  section,  FSO  rate  FSE,  etc.):  (Circle  one  value  in  every  cell  under  the  plan-monitor-direct 

columns) 

PLAN 

MONITOR 

DIRECT 

a.  Reconnaissance  operations 

0  1  2  3  4  5 

0  1  2  3  4  5 

0  1  2  3  4  5 

b.  Parallel  planning 

0  1  2  3  4  5 

0  1  2  3  4  5 

0  1  2  3  4  5 

c.  Information  management 

0  1  2  3  4  5 

0  1  2  3  4  5 

0  1  2  3  4  5 

d.  Integration  of  fires 

0  1  2  3  4  5 

0  1  2  3  4  5 

0  1  2  3  4  5 

e.  Accelerated  decision  making 

0  1  2  3  4  5 

0  1  2  3  4  5 

0  1  2  3  4  5 

1 5.  Rate  the  staff  processes  of  your  element  in  late  March  99  (Cdr/XO  rate  entire  staff,  S3  rate 
S3  Staff  section,  etc.):  Circle  one  value  in  every  cell  under  the  plan-monitor-direct  columns) 

PLAN 

MONITOR 

DIRECT 

a.  Reconnaissance  operations 

0  1  2  3  4  5 

0  1  2  3  4  5 

0  1  2  3  4  5 

b.  Parallel  planning 

0  1  2  3  4  5 

0  1  2  3  4  5 

0  1  2  3  4  5 

c.  Information  management 

0  1  2  3  4  5 

0  1  2  3  4  5 

0  1  2  3  4  5 

d.  Integration  of  fires 

0  1  2  3  4  5 

0  1  2  3  4  5 

0  1  2  3  4  5 

e.  Accelerated  decision  making 

0  1  2  3  4  5 

0  1  2  3  4  5 

0  1  2  3  4  5 

c-io 


To  No 
Extent 


Slight 

Extent 


16a.  Considering  the  improvement  in 
staff  processes  from  Jun  98  to  Mar  99, 
indicate  the  extent  to  which  the  follow¬ 
ing  contributed  to  the  improvement: 


1)  BSTS  Courses _ 

2)  COBRAS  Vignettes 


3)  Janus  Exercise  (Oct  98' 


4)  Performance  Objectives  Package 

5)  Leaders  Training  Program 


6)  Bde/Bn  Staff  Exercise  (Jan  99) 

7)  Gauntlet _ 

8)  BBS  Exercise  (Mar  99 


9)  Other 

10)  Other _ _ _ __ 

11)  Other _ 


16b.  Please  explain  how  key  items  contributed  to  the  improvement: 


Circle  one  for  each  item: 


Moderate 

Extent 


Great 

Extent 


To  a  Very 
Great  Extent 


0 

1  2  3 

o 

1  2  3 

0 

1  2  3 

0 

1  2  3 

i  o  ! 

1  2  3 

i  o  i 

1  2  3 

mm 

!  1  2  3 

0 

1  2  3 

1 7a.  Indicate  the  extent  to  which  the 
Battle  Staff  Training  System  (BSTS) 
courses  provided  knowledge,  skills, 
and  abilities  directly  relevant  to: 

(1)  The  RSOI  training. 


2)  The  force-on-force  trainin 


(3)  The  live  fire  training. _ 

17b.  Please  explain: 


Circle  one  for  each  item: 

Don’t 

To  No 

Slight 

Moderate 

Great 

To  a  Very 

|  Know 

Extent 

Extent 

Extent 

Extent 

Great  Extent 

0 

i 

2 

3 

4 

5 

1  0 

i 

2 

3 

4 

5 

i  o 

i 

2 

3 

4 

5 

C-ll 


1 8a.  Indicate  the  extent  to  which  the 


COBRAS  Vignettes  provided  know¬ 
ledge,  skills,  and  abilities  directly 
relevant  to: _ 

(1)  The  RSOI  training. _ 

(2)  The  force-on-force  training. 

(3)  The  live  fire  training. _ 

1 8b.  Please  explain: 


Don’t 

Know 


0 

0 

0 


1 9a.  Indicate  the  extent  to  which  the 
Janus  exercise  (Oct  98)  provided 
knowledge,  skills,  and  abilities  directly 
relevant  to:  _ 

(1)  The  RSOI  training. _ 

(2)  The  force-on-force  training. _ 

(3)  The  live  fire  training. _ 

1 9b.  Please  explain: . 


Don’t 

Know 


0 

0 

0 


20a.  Indicate  the  extent  to  which  the 
Brigade/Battalion  Staff  Exercise  (Jan 
99)  provided  knowledge,  skills,  and 
abilities  directly  relevant  to: _ 


(1)  The  RSOI  training. 


(2)  The  force-on-force  training. 


(3)  The  live  fire  training. 


20b.  Please  explain: 


Don’t 

Know 


0 

0 

0 


Circle  one  for  each  item 


To  No 

Slight 

Moderate 

Great 

To  a  Very 

Extent 

Extent 

Extent 

Extent 

Great  Extent 

i 

2 

3 

4 

5 

i 

2 

3 

4 

5 

i 

2 

3 

4 

5 

_  Circle  one  for  each  item: 

To  No  Slight  Moderate  Great  To  a  Very 

Extent  Extent  Extent  Extent  Great  Extent 

1  2  3  4  5 

1  2 _ 3  4 _ 5 

1  2  3  4  5 


_Circleone for^each  item: 

To  No  Slight  Moderate  Great  To  a  Very 

Extent  Extent  Extent  Extent  Great  Extent 

1  2  3  4  5 

1  2  3  4 _ 5 

1  2  3  4  5 


Impact  on  NTC  Performance 


22.  Indicate  the  extent  to  which 

Circle  one  for  each  item: 

your  training  with  the  Force  XXI 
products: 

Don’t 

Know 

To  No 
Extent 

Slight 

Extent 

Moderate 

Extent 

Great 

Extent 

To  a  Very 
Great  Extent 

a.  Helped  you  understand  your 
commander’s  expectations. 

0 

i 

2 

3 

4 

5 

b.  Helped  you  meet  your  command¬ 
er’s  information  requirements. 

0 

i 

2 

3 

4 

5 

c.  Helped  you  support  your  com¬ 
mander’s  decision-makinq  process. 

0 

i 

2 

3 

4 

5 

d.  Improved  your  situational 
awareness  skills. 

0 

i 

2 

3 

4 

5 

e.  Enhanced  your  unit’s  synchroniza¬ 
tion  of  combat,  CS,  and  CSS  assets. 

0 

i 

2 

3 

4 

5 

23a.  Indicate  the  extent  to  which 
the  simulation  exercises  you 
participated  in  (Janus,  BBS): 

Circle  one  for  each  item: 

To  No 
Extent 

Slight 

Extent 

Moderate 

Extent 

Great 

Extent 

To  a  Very 
Great  Extent 

(1)  Helped  you  orient  to  NTC  terrain. 

0 

1 

2 

3 

4 

5 

(2)  Helped  you  do  terrain  analysis. 

0 

1 

2 

3 

4 

5 

(3)  Helped  you  visualize  the  plans. 

0 

1 

2 

3 

4 

5 

(4)  Helped  you  monitor  the  battle. 

o  i 

1 

2 

3 

4 

5 

23b.  Please  explain  how  the  simulation  exercises  helped: 


Impact  on  Unit  Training  Program 


26.  In  light  of  your  NTC  experi- 

Circle  one  for  each  item: 

ence,  to  what  extent  did  your  train¬ 
ing  with  the  Force  XXI  products: 

Don’t 

Know 

To  No 
Extent 

Slight 

Extent 

Moderate 

Extent 

Great 

Extent 

To  a  Very 
Great  Extent 

a.  Help  your  unit  meet  its  staff  training 
goals? 

0 

i 

2 

3 

4 

5 

b.  Make  your  unit  better  prepared  for 
its  NTC  rotation? 

0 

i 

2 

3 

4 

5 

c.  Save  training  time  for  your  unit? 

0 

i 

2 

3 

4 

5 

d.  Save  training  funds  for  your  unit? 

0 

i 

2 

3 

4 

5 

e.  Enhance  your  unit’s  deployability? 

0 

i 

2 

3 

4 

5 

f.  Help  your  unit  improve  its  SOPs? 

0 

i 

2 

3 

4 

5 
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ISAT  STRUCTURED  INTERVIEW  GUIDE 


BBSE  Execution 


PURPOSE 

This  form  provides  instructions  for  ISAT  Team  members  when  they  conduct  interviews 
with  the  training  audience,  Exercise  Director,  COBRAS  Coordinator,  BLUFOR 
Controller,  roleplayers,  interactors,  observers,  and  site  staff  regarding  BBSE  training. 


INSTRUCTIONS 

A.  The  purpose  of  the  interview  is  to  follow  up  important  themes  noted  while  observing 
unit  personnel  as  they  execute  the  BBSE. 

B.  Review  the  attached  list  of  questions  before  the  interview  starts.  Select  5-7 
questions  dealing  with  key  issues  that  surfaced  during  preparation  or  execution  of 
training.  Plan  to  spend  about  10  minutes  on  each  question. 

C.  Use  the  questions  as  a  guide.  When  unique  issues  important  to  the  group  arise, 
make  time  to  discuss  them. 

D.  Establish  a  relaxed,  friendly  atmosphere  at  the  outset.  Tell  the  group  why  their  input 
is  important.  State  how  long  you  expect  the  session  to  last. 

E.  If  possible,  have  another  Team  member  take  notes.  Capture  the  name,  duty 
assignment,  and  unit  of  each  group  member.  Take  careful  notes — when  in  doubt, 
write  it  down. 

F.  Use  a  tape  recorder  to  obtain  an  audio  record  for  backup  purposes.  At  the  outset, 
ask  if  anyone  objects  to  this. 

G.  Be  sure  you  understand  what  the  participants  are  saying.  Ask  for  clarification  when 
needed. 

H.  Be  an  unbiased  facilitator.  Avoid  leading  the  trainers  with  your  expectations. 

I.  Listen  to  comments  and  suggestions  with  an  open  mind.  Avoid  defending  our 
products. 

J.  Try  to  keep  the  group  focused  on  the  task  at  hand.  Avoid  digressions. 

K.  Try  to  limit  the  session  to  90  minutes  or  less;  take  your  cues  from  the  participants. 

L.  Organize  your  notes  and  enter  them  into  a  file  as  soon  as  you  can.  Forward  them  to 
Bruce  within  one  week  after  the  end  of  the  exercise. 

M.  Take  questions  about  the  use  of  this  form  to  Bruce  (502-352-5200). _ 
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TRAINING  AUDIENCE  (Primary  &  Secondary) 


Pre-Exercise  Preparation 

•  What  actions  did  you  take  after  the  December  train-up  to  prepare  for  the  exercise? 
How  much  time  did  you  spend  on  each  action?  (Consider:  reviewing  Training 
Audience  Guide  and  other  materials,  copying  overlays,  reviewing  FMs,  etc.) 

•  What  would  you  do  differently  during  preparation  for  the  exercise  next  time? 
(Consider:  access  to  materials,  train-up,  time  available,  etc.) 


Exercise  Execution 

•  What  problems  did  you  encounter  in  executing  the  exercise?  (Consider:  getting 
started,  physical  facilities,  tactical  commo,  materials  provided,  quality  of  orders  and 
messages  from  EXCON,  coaching,  AARs,  training  schedule,  etc.) 

•  How  effective  were  the  coaching  and  AARs?  How  could  the  performance 
assessment  and  feedback  procedures  be  improved? 

•  What  other  execution  materials  (e.g.,  job  aids,  checklists)  would  you  like  to  have? 


Contributions  to  Unit  Training  Program 

•  How  might  training  with  the  exercise  benefit:  (a)  subsequent  homestation  collective 
training;  (b)  unit  readiness  for  the  NTC? 

•  How  might  this  training  help  your  unit  develop/improve  its  SOPs? 

•  How  valuable  would  your  unit  find  similar  exercises  for  future  training?  How  can  the 
exercises  be  shaped  for  maximum  training  benefit? 


Implementation  Issues 

•  What  obstacles  do  you  envision  if  your  unit  trains  with  similar  exercises  in  the 
future?  What  can  be  done  to  overcome  those  obstacles? 

•  What  could  be  done  to  better  enable  the  unit  to  execute  this  exercise  in  the  future? 
What  assistance  would  be  needed  from  outside  the  division? 


How  to  Enhance  the  BBSE? 

•  How  would  you  improve  the  training  materials  and  procedures?  (Consider:  doctrinal 
consistency,  schedule  of  events,  site  support,  etc.) 

•  What  enhancements  are  needed  to  meet  training  requirements  for  digital 
operations? 


ROLEPAYERS  AND  INTERACTORS 

Pre-Exercise  Preparation 

•  What  actions  did  you  take  after  the  December  train-up  to  prepare  for  the  exercise? 
How  much  time  did  you  spend  on  each  action?  (Consider:  preparing  and 
organizing  job  aids,  copying  materials,  reviewing  the  TACSOP,  etc.) 

•  What  would  you  do  differently  during  preparation  for  the  exercise  next  time? 
(Consider:  access  to  materials,  train-up,  time  available,  etc.) 

•  What  other  preparation  activities  should  have  been  accomplished  prior  to 
STARTEX? 

Exercise  Execution 

•  What  problems  did  you  encounter  in  executing  the  exercise?  (Consider:  getting 
started,  physical  facilities,  tactical  commo,  materials  provided,  workstation 
operations,  workload,  assistance  from  ISAT  or  Site  Teams,  etc.) 

•  How  well  prepared  were  you  when  the  exercise  started?  In  what  areas  did  you  find 
yourselves  under-prepared? 

•  What  other  execution  materials  (e.g.,  job  aids,  checklists)  would  you  like  to  have? 

Contributions  to  Unit  Training  Program 

•  How  do  you  expect  to  benefit  from  your  participation  in  this  exercise?  How  might 
your  unit  benefit? 

•  How  might  this  training  help  your  unit  develop/improve  its  SOPs? 

•  How  valuable  would  your  unit  find  similar  exercises  for  future  training?  How  can  the 
exercises  be  shaped  for  maximum  training  benefit? 

Implementation  Issues 

•  How  important  was  the  external  assistance  from  the  ISAT  and  Site  Teams  in 
enabling  you  to  effectively  support  the  exercise?  Which  aspects  were  critical? 

How  to  Enhance  the  BBSE? 

•  How  could  the  materials  you  worked  with  be  improved?  What  new  materials  would 
you  like  to  have? 

•  What  enhancements  would  be  needed  to  meet  training  requirements  for  digital 
operations? 


Pre-Exercise  Preparation 

•  What  actions  did  you  take  after  the  December  train-up  to  prepare  for  the  exercise? 


How  much  time  did  you  spend  on  each  action?  (Consider:  reviewing  the  EXCON 
Roleplayer  Guide  and  other  materials,  preparing  and  organizing  job  aids,  etc.) 

•  How  would  you  improve  the  advance  materials  and  procedures?  (Consider:  access 
to  materials,  train-up,  time  available,  etc.) 

•  What  other  preparation  activities  should  have  been  accomplished  prior  to 

STARTEX? _ 

Exercise  Execution 

•  What  problems  did  you  encounter  in  supporting  the  brigade  staff?  (Consider:  getting 
started,  physical  facilities,  tactical  commo,  maps,  materials  provided,  workstation 
operations,  workload,  coordination,  etc.) 

•  How  well  prepared  were  you  when  the  exercise  started?  In  what  areas  did  you  find 
yourselves  under-prepared? 

•  What  activities  were  most  difficult  to  perform?  Why?  (Consider:  sending  prepared 
messages,  responding  to  brigade  requests  for  information,  monitoring  the 
simulation,  preparing  Intel  updates,  operating  the  workstations,  etc.) 

Contributions  to  Unit  Training  Program 

•  How  do  you  expect  to  benefit  from  your  participation  in  this  exercise?  How  might 
your  unit  benefit? 

•  How  might  this  training  help  your  unit  develop/improve  its  SOPs? 

•  How  valuable  would  your  unit  find  similar  exercises  for  future  training?  How  can  the 
exercises  be  shaped  for  maximum  training  benefit? 

Implementation  Issues 

•  How  important  was  the  external  assistance  from  the  I  SAT  and  Site  Teams  in 
enabling  you  to  effectively  support  the  exercise?  Which  aspects  were  critical? 

How  to  Enhance  the  BBSE? 

•  How  would  you  improve  the  training  materials  and  procedures?  (Consider:  quality 
of  tactical  materials,  staffing,  coordination  procedures,  site  support,  etc.) 

•  What  enhancements  would  be  needed  to  meet  training  requirements  for  digital 
operations? 
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OBSERVERS  v 

Pre-Exercise  Preparation 

•  What  actions  did  you  take  after  the  December  train-up  to  prepare  for  the  exercise? 
How  much  time  did  each  action  take?  (Consider:  reviewing  Observer  Guide  and 
Performance  Objectives,  preparing  the  observation  plan,  developing  AAR  schedule, 
reviewing  the  pertinent  TACSOP,  tactical  familiarization,  etc.) 

•  What  problems  did  you  encounter  in  obtaining  copies  of  each  Performance 
Objective’s  observation  guide?  (Consider:  availability  of  materials  in  Apx  C  of  the 
Observer  Guide,  duplication  of  materials,  etc.) 

•  What  would  you  do  differently  during  preparation  for  the  exercise  next  time? 
(Consider:  scheduling,  additional  materials,  train-up,  time  available,  etc.) 

Exercise  Execution 

•  What  problems  did  you  encounter  in  serving  as  an  Observer?  (Consider: 
implementing  the  observation  plan,  coaching,  preparing  and  presenting  AARs, 
workload  issues,  physical  facilities,  tactical  commo,  admin  commo,  etc.) 

•  How  easy  was  it  to  use  the  observation  guide  for  each  Performance  Objective  during 
the  exercise?  (Consider:  basic  instructions,  organization  of  information,  bulk, 
clarity,  ease  of  recording  performance,  etc.) 

•  How  effective  were  the  coaching  and  AAR  procedures?  How  could  the  performance 
assessment  and  feedback  procedures  be  improved? 

•  How  do  you  feel  about  the  take-home  summary  reports?  How  would  you  change 
them? 

Contributions  to  Unit  Training  Program 

•  How  might  training  with  the  exercise  benefit:  (a)  subsequent  homestation  collective 
training;  (b)  unit  readiness  for  the  NTC  rotation? 

•  How  valuable  would  the  unit  find  similar  exercises  for  future  training?  How  can  the 
exercises  be  shaped  for  maximum  training  benefit? 

How  to  Enhance  the  BBSE? 

•  How  would  you  improve  the  observation  and  feedback  procedures?  (Consider: 

Observer  staffing,  observation  materials  and  procedures,  coaching,  AAR 
procedures,  seguence  of  events,  administrative  coordination,  etc.) _ 
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CONTROL  PERSONNEL 

(Exercise  Director,  COBRAS  Coordinator,  BLUFOR  Controller) 


Pre-Exercise  Management  and  Preparation 

•  What  actions  did  you  or  your  representative  take  after  the  December  train-up  to 
prepare  for  the  exercise?  How  much  time  did  each  action  take?  (Consider: 
reviewing  Exercise  Guide,  coordinating  facilities  and  participants,  obtaining  maps 
and  supplies,  copying  and  distributing  materials,  etc.) 

•  What  would  you  do  differently  during  preparation  for  the  exercise  next  time? 
(Consider:  scheduling,  coordinating  facilities  and  participants,  distributing  materials, 
train-up,  time  available,  etc.) 


Exercise  Execution 

•  What  problems  did  you  observe  in  executing  the  exercise?  (Consider:  getting  the 
exercise  started,  monitoring  and  controlling  the  exercise,  observing  performance, 
preparing  and  presenting  AARs,  EXCON  and  OPFOR  staffing,  physical  facilities, 
tactical  commo,  admin  commo,  training  schedule,  workload  issues,  etc.) 

•  What  coordination  procedures  did  you  use  (e.g.,  White  Cell)?  How  well  did  the 
procedures  work  for  surfacing  and  resolving  issues?  What  would  you  change? 

•  What  other  execution  materials  (e.g.,  job  aids,  checklists)  would  you  like  to  have? 


Contributions  to  Unit  Training  Program 

•  How  might  training  with  the  exercise  benefit:  (a)  subsequent  homestation  collective 
training;  (b)  unit  readiness  for  the  NTC  rotation? 

•  How  valuable  would  the  unit  find  similar  exercises  for  future  training?  How  can  the 
exercises  be  shaped  for  maximum  training  benefit? 


Implementation  Issues 

•  What  could  be  done  to  better  enable  the  unit  to  execute  this  exercise  in  the  future? 
What  obstacles  would  have  to  be  overcome?  What  external  assistance  would  be 
needed? 


How  to  Enhance  the  BBSE? 

•  How  could  the  TSP  be  improved  to  better  meet  anticipated  training  needs? 
(Consider  doctrinal  consistency,  emerging  unit  missions,  training  environment,  etc.) 

•  What  enhancements  are  needed  to  meet  training  requirements  for  digital 
operations? 
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SITE  STAFF 

Pre-Exercise  Preparation 

•  What  actions  did  you  take  after  the  December  train-up  to  prepare  for  the  exercise? 
How  much  time  did  you  spend  on  each  action?  (Consider:  reviewing  Site  Manager 
Guide  and  other  materials,  tactical  familiarization,  workstation  and  commo  set-up, 
loading  and  testing  BBS  files,  coordinating  with  the  unit  or  ISAT  Team,  etc.) 

•  What  would  you  do  differently  during  preparation  for  the  exercise  next  time? 
(Consider:  coordination,  set-up,  train-up,  time  available,  facilities,  etc.) 

•  What  other  preparation  activities  should  have  been  accomplished  prior  to 
STARTEX?  (Consider:  site  activities,  unit  activities,  coordination,  etc.) 

Exercise  Execution 

•  What  problems  did  you  encounter  in  supporting  the  exercise?  (Consider:  getting 
started,  physical  facilities,  materials  provided,  workstation  operations,  site  staffing, 
workload,  etc.) 

•  What  assistance  did  you  provide  to  the  interactors  and  roleplayers?  In  what  areas 
did  they  have  a  lot  of  difficulty? 

•  What  other  execution  materials  (e.g.,  job  aids,  checklists)  would  you  like  to  have? 

•  How  would  you  recommend  changing  the  staffing  of  the  workstation  cells? 
(Consider:  types  of  personnel,  number  of  personnel,  allocation  of  functions,  etc.) 

•  What  problems  resulted  from  brigade  and  battalion  personnel  training  in  the  same 
exercise?  (Consider:  eavesdropping,  unrealistic  interaction,  game-playing,  etc. 

Implementation  Issues 

•  How  many  site  staff  members  should  be  on-hand  during  execution  of  this  exercise? 
Where  should  they  be  located? 

•  What  additional  site  support  should  the  unit  have  in  executing  this  exercise  in  the 
future?  (Consider:  facilities,  equipment,  staffing,  administrative  needs,  etc.) 

How  to  Enhance  the  BBSE? 

•  How  could  the  materials  you  worked  with  be  improved?  What  other  materials  would 
you  like  to  have? 

•  What  enhancements  would  be  needed  to  meet  training  requirements  for  digital 

operations? _ 
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ISAT  OBSERVER’S  GUIDE 
BBSE  Execution 


Observer: _  Assignment: 


Date: 
Unit:  _ 


PURPOSE 

This  guide  provides  instructions  for  ISAT  data  collectors  when  they  observe  the  BBSE 
execution  at  the  Battle  Simulation  Center.  It  lists  questions  of  interest  and  organizes  the 
note-taking. 


INSTRUCTIONS 

A.  Keep  this  guide  with  you  as  you  observe  the  exercise.  Use  it  to  structure  your 
observation  activities  and  note-taking.  There  is  only  one  form  for  ISAT  observers. 

B.  Review  the  attached  list  of  questions  at  the  outset.  They  are  only  a  guide.  Be  sure 
to  capture  all  worthwhile  information,  whether  it’s  listed  in  this  guide  or  not. 

C.  Observe  the  elements  you’re  responsible  for — watch,  listen,  query.  Chat  with  the 
participants  during  breaks.  Feel  free  to  comment  on  other  elements — just  be  sure  to 
state  to  whom  your  comments  are  referring. 

D.  Be  sure  you  understand  what  the  participants  are  doing  and  saying.  Ask  for 
clarification  when  needed,  but  don’t  interrupt  something  important. 

E.  Rotate  the  focus  of  your  attention.  When  something  of  special  interest  occurs,  try  to 
shift  to  that. 

F.  Listen  to  comments  and  suggestions  with  an  open  mind.  Avoid  defending  our 
products. 

G.  Record  your  observations  and  the  participants’  comments  by  writing  on  this  form. 
When  in  doubt,  write  it  down.  Use  extra  paper  as  needed. 

H.  Allow  the  trainers  and  participants  to  use  the  resources  in  the  TSP.  Your  job  is  to 
monitor,  not  “save”  the  unit  or  force  the  products  to  look  good. 

I.  Do  not  direct  or  interfere  with  the  activities  of  the  training  audience  or  trainers. 

J.  Give  your  notes  to  Dave  Pratt  at  the  end  of  the  exercise. 

K.  Take  questions  about  the  use  of  this  guide  to  Bruce  Leibrecht,  502-352-5200. _ 
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Pre-Exercise  Activities 


Question 

Comments 

What  did  the  personnel  in  your 
element  do  to  prepare  since 
the  train-up?  How  much  time 
did  they  take? 

What  materials  do  the  partici¬ 
pants  bring  to  the  exercise? 

What  materials  are  they 
missing? 

What  confusion  is  there  among 
participants,  trainers,  etc. 
about  what  to  expect? 

Does  the  unit  have  a  detailed 
training  schedule?  What 
events  are  planned? 

How  would  you  rate  the 
readiness  of  your  elements  at 
the  start  of  training?  What 
shortfalls  do  you  see? 

Other  observations: 

Exercise  Execution 


Question 

Comments 

Who  serves  as: 

•  Exercise  Director? 

•  COBRAS  Coordinator? 

BLUFOR  Controller? 

Have  these  changed  since  the 
train-up? 

Who  supervises  or  directs  the 
activities  in  your  element? 

How  does  he  perform  his  job? 
How  much  of  the  time  is  he  on¬ 
site? 

What  significant  departures 
from  the  published  schedule 
occur? 

Is  your  element  staffed  as 
specified  in  the  TSP?  If  not, 
who  is  missing?  Are  there  any 
stand-ins? 

Do  personnel  stay  for  the 
entire  exercise? 

Interactor/Roleplayer  refresher 
training:  What  refresher 
training  occurs?  Who  delivers 
the  instruction? 

What  misunderstandings  are 
apparent  re:  training 
objectives,  roles  and 
responsibilities,  etc.? 

How  do  the  Observers  perform 
their  jobs?  Do  they  use  an 
observation  plan?  Do  they  use 
the  assessment  guide  for  each 
Performance  Objective? 

How  do  the  Observers  depart 
from  the  TSP? 

What  kinds  of  requests  for 
assistance  do  you  receive  from 
personnel  in  your  element? 

What  problems  occur  during 
the  course  of  the  exercise? 

Why  do  the  problems  occur? 
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How  do  they  impact  the 
training? 

Are  important  problems 
addressed  by  the  Exercise 

Director? _ 

Do  personnel  in  you  element 
appear  to  have  the  appropriate 
skills  to  carry  out  their  tasks? 
Describe  any  apparent  short- 

comings. _ 

What  facilities  problems 
occur?  (Include  observations 
regarding  the  BBS 

environment.) _ 

What  suggestions  do  you  hear 
for  improving  the  training 
materials  and  procedures? 

How  do  the  personnel  in  your 
element  feel  about  the  value  of 

the  training? _ 

Comment  on  any  apparent 
difficulties  related  to  using  BBS 

to  drive  the  exercises. _ 

Do  you  observe  specific 
behaviors  or  events  which 
might  degrade  the  quality  of 
training?  Describe  such 

observations. _ 

Other  observations: 
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AAR  Activities 


Question 

Comments 

Who  leads  the  AARs  for  your 
element  (Bde/TF)?  How  does 
he  prepare  for  the  AARs? 

How  much  time  does  he  take? 

What  steps  are  followed  in 
conducting  the  AARs?  How 
do  they  differ  from  the  AAR 
procedures  recommended  in 
the  TSP? 

How  long  does  each  AAR 
take?  Is  it  enough  time? 

Does  the  training  audience 
end  up  with  a  clear 
understanding  of  strengths  and 
weaknesses?  If  not,  why? 

Do  the  Cdr  and  his  staff 
appear  satisfied  with  the 

AARs?  If  not,  explain. 

What  are  two  keys  for 
improving  the  AARs? 

Other  observations? 

Appendix  D 
FXXI  Training  Program 
Fielding-Update-Sustainment  Strategy 


A  Briefing 

Presented  to 
COL  Blankmeyer,  DTDD 
18  March  1999 


FXXI-Training  Program 
Fielding-Update-Sustainment 
Strategy 


Over  the  past  five  years,  the  Force  XXI  Training  Program  has  been  responsible 
for  the  development  of  a  variety  of  training  products  for  combined  arms  forces. 
The  training  products  have  been  used  in  a  limited  number  of  units,  but  are  not 
yet  widely  fielded.  Furthermore,  no  mechanism  is  in  place  to  ensure  that  the 
products  are  upgraded  in  response  to  changes  in  doctrine,  organizations,  and 
equipment. 
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Purpose 


♦  To  provide  DTDD  and  the  Force  XXI  -  Training 
Program  with  a  strategy  for  fielding,  updating, 
and  sustaining  the  products  developed  over 
the  last  five  years 

♦  To  provide  an  estimate  of  the  effort  required  to 
update  the  programs 
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Force 
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The  purpose  of  this  briefing  is  describe  the  requirements  for  fielding,  updating, 
and  sustaining  the  products,  as  well  as  to  provide  an  estimate  of  the  update 
effort  in  both  staff  months  and  calendar  months.. 
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Agenda 


♦  Background 

♦  The  Strategy 

♦  Fielding 

♦  Updating  Current  Products 

♦  Supporting  the  Force 

♦  Surge  Effort  Considerations 

♦  Sustaining  the  Products 


3/18/99' 


Force 


The  agenda  includes: 

•  The  background  on 

-Products 

-the  stepping  stone  approach  to  using  the  products 

-feedback  received  from  the  force  during  the  trials  of  these 
products  and  during  the  ISAT  project 

•  The  strategy  for  initially  fielding  and  simultaneously  updating  the 
products 

•  An  approach  to  both  low-level  and  surge-level  support  for  units 

•  Discussion  of  the  on-going  need  for  sustainment 
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Background 


Products 


Product  Developed 

Battle  Staff  Training  System  (BSTS)  1994-1996 

Bde  Common  Core  modified  to  CBI  1 998 

Small  Group  Exercises  (SGE)  1 994-1996 

I  Format  of  earlier  exercises  modified  1 998 

|  Brigade  Staff  Exercise  (BSE)  1994-1996 

|  Bde/Bn  Staff  Exercise  (BBSE)  1997-1998 
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The  Battle  Staff  Training  System  (BSTS)  -series  of  modules  for  brigade  and  battalion  staffs 
that  provide  individual  instruction,  practice,  and  comprehensive  testing  on  staff  tasks  and 
processes.  Comprises  both  computer-based  and  paper-based  materials,  and  also  a  training 
management  system.  Completed  in  1996,  partially  updated  in  1998  to  convert  one  set  of 
materials  to  completely  computer-based  mode. 

Small  Group  Exercises  (SGE)  -  24  self-contained  vignette  exercises  for  the  brigade  staff. 
Focuses  on  selected  members  of  the  staff  reacting  to  a  well-defined  problem  in  a  scenario 
setting.  Problems  are  drawn  from  such  sources  as  NTC  and  other  CTC  trends  analyses  and 
first-hand  experience  of  NTC  O/C).  20  of  the  exercises  require  no  simulation  support;  the 
other  four  use  Janus  or  the  Brigade  and  Below  Battle  Simulation  (BBS).  Completed  in  1 996, 
and  in  1998  modified  slightly  to  increase  usability. 

The  Brigade  Staff  Exercise  (BSE)  -  A  BBS-driven  structured  exercise  for  the  commander  and 
staff  of  a  combined  arms  brigade.  Three  missions  are  included.  The  BSE  focuses  on  staff 
coordination  and  use  of  all  assets  throughout  mission  planning,  preparation,  execution,  and 
consolidation  and  reorganization.  The  BSE,  which  was  completed  in  1996,  provides  basic 
“crawl”  level  staff  practice  opportunities. 

The  Brigade  and  Battalion  Staff  Exercise  (BBSE)  -  A  BBS-driven  structured  exercise  for  the 
commander,  staff,  and  one  to  three  battalions  of  a  combined  arms  brigade.  The  BBSE 
incorporates  many  of  the  features  of  a  CTC  rotation,  including  24-hour  operations;  deployed 
command  posts  (CPs),  concurrent  planning  and  execution,  and  a  robust  OPFOR.  It  focuses 
on  bringing  the  brigade  to  a  level  where  they  can  derive  maximum  benefit  from  a  CTC 
rotation.  The  BBSE  was  completed  in  1998. 

As  we  already  know,  those  products  delivered  prior  to  1998  are  outdated,  since  Army  doctrine 
changed  in  the  1997/1998. 
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Background 


A  Stepping  Stone  Approach 

for  Brigade  Training 
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This  slide  shows  the  stepping  stone  approach  to  using  the  force  XXI-TP 
products. 

It  doesn’t  require  every  unit  to  start  from  the  bottom,  it  just  shows  a  user  where 
the  products  fit  into  a  training  model.  In  fact  several  versions  of  this  slide  have 
been  developed  and  in  some  you  will  see  arrows  going  from  one  step  to  another 
showing  that  a  unit  can  use  the  products  at  any  time,  depending  on  their  specific 
needs. 

The  products  complement  live  training,  such  as  field  training  exercises  (FTXs) 
and  rotations  to  the  Combat  Training  Centers  (CTCs).  They  are  not  designed  as 
single-implementation  exercises,  but  as  tools  that  can  be  used  repeatedly  to 
enhance  skill  retention  over  time  between  live  training  exercises,  to  counter  the 
effects  of  personnel  turbulence,  and  to  reinforce  lessons  learned  in  live  training 
exercises. 
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Background 

Feedback  from  the  Force 


♦  Good  Concept ...  Needs  to  be  Fielded 

...  Must  remain  viable 

♦  Some  Dated  Material ...  Updates  required 

♦  Very  Complex  ...  Need  help /Support 


Feedback  from  the  field  is  that  the  products  are  useful,  but  they  must  be  updated 
as  doctrine  and  TTP  changes. 


They  also  have  a  lot  of  material  associated  with  them  and  are  considered 
somewhat  complex.  As  a  result,  they  are  much  more  likely  to  be  used  if,  when 
delivered  to  the  forces,  some  type  of  train-the-trainer  team  assists  in  the 
implementation  of  the  products. 


....needs  to  be  fielded  --  unit  leaders  that  saw  the  displays  during  previous 
Armor  Conferences 

....must  remain  viable  --  COL  (P)  Dyer,  DGC  (M),  Ft.  Riley 
....dated  material  -  LTC  Niedringhaus,  7ATC;  &  COL  Jones,  16th  CAV 
....very  complex  --  trial  units  and  supporting  BSCs 
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Background 


Issues 


FIELD 

UPDATE 

SUSTAIN 

*  Educate 

*  Concurrent 

*  Ongoing 

*  Deliver 

w/Fielding 

*  Adapt  j 

*  Implement 

*  Ongoing 
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Focus  on  the  three  needs  that  underlie  the  strategy. 


Fielding  -  As  we  stated  earlier,  we  can’t  just  deliverthe  products  to  the  force. 
We  need  to  field  {hem.  Fielding  entails  three  components,  education,  delivery, 
and  implementation.  If  we  don’t  include  all  three  in  our  fielding  strategy,  we’ll 
run  into  problems  when  the  units  try  to  use  the  products. 

Updating  -  This  is  needed  because  we’ve  changed  our  doctrine  since  some  of 
the  products  were  delivered  to  DTDD,  and  we’ve  changed  how  we  think  about 
and  use  technology.  Naturally,  these  need  to  be  updated.  The  issue  here  is  do 
we  update  first  and  then  deliver  or  do  we  deliver  what  we  have  now,  knowing 
that  parts  of  the  products  are  outdated  and  tell  the  field  that  we’ll  get  them 
revised  products  later.  Either  way,  our  decision  must  be  part  of  the  education 
process  of  fielding. 

Sustaining  -  If  these  products  are  going  to  remain  VIABLE,  as  COL  Dyer  stated, 
me  must  have  a  plan  to  update  them  as  our  doctrine  and  TTP  changes.  This 
will  be  an  ongoing  challenge.  The  products  must  also  be  adaptable  to  unit 
needs 
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Strategy  for  Fielding  and 
Sustainment  of  Products 


FIELD 


SUSTAIN  \ 
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This  slide  depicts  our  strategy.  We’ll  use  this  picture  as  we  describe  each  part 
of  the  strategy. 

As  shown,  initial  fielding  gradually  transitions  to  ongoing  sustainment.  Similarly, 
initial  updates  are  succeeded  by  a  process  of  continuing  updates.  Feedback  and 
lessons  learned  during  fielding  are  used  to  make  the  initial  updates,  which  are 
then  delivered  and  incorporated  into  the  already  fielded  products.  This  process 
continues  over  time:  Units  use  the  products  and  provide  suggestions  or 
concerns,  and  developers  use  the  information  to  make  continual  improvements. 
Although  solutions  for  each  part  of  the  process  can  be  planned  and  executed 
separately,  all  three  needs  must  be  addressed  in  order  for  any  solution  to  be 
effective. 
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Fielding  has  three  components:  education,  delivery  and  implementation. 
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Fielding 


Educate  the  Force 


♦  How: 

♦  Initial  Orientation  (Conference) 

♦  Targeted  Briefings  (On-site) 

♦  Implementation  Support  (On-site) 

♦  To  Whom: 

♦  Installation  Leadership 

♦  Simulation  Centers 


First  we  have  to  develop  a  plan  for  educating  the  users  of  the  products. 


Our  recommendation  starts  with  a  large  conference,  like  the  armor  conference, 
the  conference  is  then  followed  targeted  briefings  to  people  who  use  the 
products  and  those  who  may  only  reference  the  products. 

-  A  typical  user  would  be  an  AC/RC  brigade  or  someone  like  a  TSB  that  is 
supporting  an  AC/RC  use  of  the  product 

-  There  are  several  other  units/agencies  in  the  force  that  may  never  use  the 
product,  but  need  to  know  about  the  product  in  order  to  make  recommendations 
to  user  units.  The  CTC’s,  BCTP  program,  and  PCC  are  examples. 

The  last  part  of  the  education  process  involves  the  on-site  train-the-trainer  type 
education  that  must  occur  at  the  using  unit  site  once  they  receive  the  materials. 


The  plan  for  educating  the  force  has  to  include  the  audience  of  the  products. 
First  we  have  to  explain  the  program  and  the  products  to  the  senior  leaders  on 
the  installations.  But  we  also  have  to  train  the  people  running  the  simulation 
centers  since  they  are  the  key  implementers  of  two  of  the  products. 
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Fielding 


Educate  the  Force 


♦  Initial  Orientation 

♦  Briefing  (DTDD) 

♦  Concept,  Products,  Strategy 

-  Delivery  Plan 

-  Implementation  Support  Plan 

♦  Product  Seminars  [DTDD  &  Contractor) 

♦  BSTS 

♦  Small  Group  Exercises  (SGE) 

♦  BSE/BBSE 

-  Exercise  Preparation  -  Exercise  Control 

-  Performance  Objectives  -  Simulation  Support 
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We  envision  the  initial  orientation  conference  to  have  two  components: 

-  The  first  component  is  an  initial  briefing,  given  by  DTDD  or  USAARMC,  that 
describes  the  program  and  the  products  and  explains  how  we’re  going  to  get 
these  products  to  the  force. 

-  The  second  component  is  a  series  of  seminars  similar  to  the  briefings  we 
provided  LTC  Niedringhaus  from  7ATC.  We  think  that  one  seminar  will  suffice 
for  the  BSTS  and  Small  Group  exercise,  but  we  may  need  to  break  the 
BSE/BBSE  workshop  down  into  these  four  categories. 


In  all,  the  initial  orientation  consists  of  a  briefing  and  six  seminars.  We  envision 
the  conference  attendees  will  attend  whichever  workshops  they  feel  are 
applicable  to  them,  thus  most  would  attend  the  BSTS  and  SGE  seminar,  but 
perhaps  only  the  BSC  folks  would  attend  the  simulation  support  seminar  for  the 
BSE/BBSE. 
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Fielding 

Educate  the  Force 


♦  Targeted  Briefings 

♦  Combat  Training  Centers 

♦  CAC 

♦  TRADOC  Schools 

♦  PCC 

♦  Staff  Training  Courses 

♦  Advance  Courses 


The  second  part  of  educating  the  force  is  targeted  briefings. 

Here  the  original  conference  briefing  would  be  modified,  tailoring  the  briefing  to 
the  audience  and  what  we  expect  their  role  to  be  in  the  training  strategy. 
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Fielding 


Educate  the  Force 


♦  Implementation  Support 

♦  On-site  Train-the-Trainer  Sessions 

♦  BSTS  &  SGE: 

-  BSTS:  Users  &  Administrators 

-  SGE:  Training  &  Support  Coordinators 

♦  BSE/BBSE: 

-  Exercise  Preparation 

-  EXCON  &  OPFOR 

-  Simulation  Support  &  Workstation  Training 

-  Observers 

-  Training  Audience 
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The  last  part  of  the  education  plan  for  fielding  is  implementation  support. 


Feedback  form  the  field  says  that  the  using  units  need  this  level  of  training  in 
order  to  fully  understand  how  and  when  (and  even  why)  to  use  the  products. 
This  training  might  include: 

-  How  each  product  is  intended  to  be  used 

-  How  each  product  can  be  modified  and  some  of  the  pitfalls  that  can  result  if 
you  aren’t  thorough.  But  to  be  thorough,  you  must  first  understand  each 
product/component  and  its  relationship  to  the  other  products  and  components. 
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Fielding 


Deliver  to  the  Force 


♦  How? 

♦  BSTS  -  ATSC/Internet 

♦  SGE  -  ATSC/Internet 

♦  BSE/BBSE  -  ATSC/Internet 

♦  To  Whom? 

♦  AC  and  RC  Units 

♦  TSBs  /  Exercise  Divisions 

♦  Installation  Simulation  Centers 

♦  TRADOC  Schools 


m 


The  second  part  of  the  fielding  strategy  is  the  delivery. 


The  strategy  in  this  area  mostly  deals  with  Who  gets  what  products.  By  “who” 
we’re  referring  to  the  type  units  listed  on  this  slide. 


One  other  question  might  be  When ? 

Our  assessment  is  that  we  need  to  get  the  current  products  out  NOW  with  a 
promise  to  update  the  problem  areas  and  deliver  the  updates  as  they  are  made. 
That’s  why  the  fielding  plan  is  important. 
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Fielding 


BSTS  Implementation 


♦  Deliver  to: 

♦  AC  and  RC  Brigades 

♦  TSBs/Exercise  Divisions 

♦  Selected  Advance  Courses 

♦  Targeted  Briefings: 

♦  CTCs 

♦  PCC 

♦  Selected  Advance  Courses 
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The  recommendation  for  delivery  of  the  BSTS  products  is  more  than  a 
broadcast  of  materials.  While  some  institutional  and  unit  trainers  would  get  the 
actual  materials,  other  groups  would  get  information  in  a  targeted  briefing. 

The  advanced  courses  we  recommend  sending  BSTS  to  are  AOAC  and  IOAC. 
However,  applicable  BSTS  courses  could  also  be  sent  to  the  FA,  ENG,  ADA, 
CHEM,  and  Signal  Officer  advanced  courses. 

In  many  cases,  course  administrators, just  get  a  briefing  to  let  participants  know 
what  they  may  see  when  they  go  to  their  next  unit. 

Another  consideration,  not  shown  on  this  slide,  is  to  give  BSTS  to  the  officer 
basic  courses  or  BNCOC. 
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Fielding 


Small  Group  Exercise 
Implementation 


The  recommendations  for  fielding  SGE  is  similar  to  the  plan  for  BSTS. 

The  only  difference  might  be  that  the  advanced  courses  other  than  AO  AC  and 
IOAC  would  be  less  likely  to  want  these  than  they  would  want  the  BSTS. 
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Fielding 

BSE/BBSE  Implementation 


♦  Deliver  to: 

♦  Installation  /  BSC 

♦  TSBs/Exercise  Divisions 

♦  Staff  Training  Courses 

♦  Educate  only: 

♦  AC  and  RC  Brigades 

♦  PCC 

♦  CTCs 

♦  Selected  Advance  Courses 


m 


The  plan  for  implementing  the  BSE  and  BBSE  is  somewhat  different.  Note  that 
instead  of  being  delivered  to  the  AC  and  RC  Brigades,  they  would  be  delivered 
to  the  installations  and  their  BSCs.  This  is  because  the  installation  and  Division 
staffs  have  a  large  role  in  supporting  this  level  of  training. 


In  the  same  manner,  we  still  have  to  tailor  a  briefing  for  the  AC  and  RC  brigades, 
so  they  understand  what  these  exercises  can  do  for  them. 


D-17 


Update  Current  Products 


SUSTAIN 


Purpose 

Production  and  Delivery  Plan 
Effort  Required 
Team  Requirements 
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Even  though  we  assert  that  fielding  can  and  should  be  ongoing  right  now,  we  all 
recognize  that  some  of  the  products  need  to  be  updated.  The  question  is  when 
and  how  to  deliver  the  updated  product. 


The  strategy  includes  starting  ASAP  with  incremental  deliveries.  The  plan  for 
doing  this  includes  the  following  considerations: 

-Purpose 

-Production  and  Delivery 
-Effort  required 
-Update  team  requirements 
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Updating 


Need  for  Updates 


♦  Align  with  current  Army  Operations 

♦  Doctrine  on  planning  and  tactics 

♦  Operational  equipment  and  weapon  systems 

♦  Division,  Brigade,  Battalion  organizations 

♦  Link  to  doctrinal  task  analytic  information  (ASAT) 

♦  Take  advantage  of  technology 

♦  Internet  capabilities 

♦  New  simulation  and  simulation  upgrades 

♦  Y2K  compliant  software 

♦  Incorporate  instructional  design  advances 

♦  Format  for  adult  learners 

♦  Engineer  with  human  factors  considerations 
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The  need  for  the  updates  is  essentially  three  fold. 

First,  there  have  been  doctrinal  changes  in  the  area  of  Army  Operations  since 
the  products  were  first  developed.  The  changes  affect  broad  areas  of  planning 
and  tactical  procedures,  equipment,  organizations,  and  task  analytic  information 

Technological  changes  are  occurring  rapidly.  The  Internet  and  simulations  offer 
new  capabilities  that  should  be  incorporated  into  our  training  approach.  At  the 
same  time,  we  need  to  go  back  into  some  of  the  products  to  ensure  Y2K 
compatibility. 

Finally,  we  are  learning  more  about  instructional  design,  human  factors  in 
computer-based  instruction,  and  adult  learning  models.  We  need  to  take  these 
into  consideration  as  we  plan  for  updating  the  products. 
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Updating 


Product  Update 
Requirements 

♦  BSTS: 

♦  Y2K  compliant  courseware 

♦  Incorporate  human  factors  in  COMPS 

♦  Incorporate  new  doctrine 

♦  Make  Internet-downloadable 

♦  SGE,  BSE,  BBSE 

♦  Incorporate  new  doctrine  in  tactical  products 

♦  Incorporate  simulation  changes  into  support  materials 

♦  Standardize  formats  and  support  materials 


BSTS  is  most  affected  by  the  Y2K  problem,  in  the  training  management  system. 
Human  factors  considerations  (usually  lumped  under  the  term  “user  friendly”) 
need  attention.  The  changes  in  doctrine  must  be  incorporated.  Finally,  the 
delivery  mode  could  and  should  transfer  to  Internet-downloadable  soon,  in  order 
to  make  the  BSTS  more  accessible  to  a  wide  range  of  users. 

For  SGE,  BSE,  and  BBSE,  we  need  to  examine  the  impact  of  doctrinal  changes 
and  simulation  changes,  and  look  at  the  TSP  design  with  an  eye  toward 
increased  user-friendliness. 
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Updating 


BSTS  Updating: 
Production  &  Delivery 


♦  BSTS 

♦  Production: 

♦  Update  courses  in  pairs  (Bde/Bn) 

♦  Priority  is  Common  Core 

♦  Bde/Bn  staff  officer  courses 

♦  Delivery: 

♦  Deliver  as  updated 

♦  Replace  IAW  fielding  strategy  (educate,  deliver, 
implement) 
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The  plan  for  updating  the  BSTS  courses  is  to  update  them  in  pairs,  starting  with 
the  Bde  and  Bn  common  core  courses  followed  the  staff  specific  courses.  Cdr, 
XO,  SI ,  etc 


Common  core  is  the  priority,  even  though  it  is  already  the  most  current  course, 
because  it  is  also  the  baseline  course.  The  staff  specific  sources  should 
assume  that  the  person  completing  this  method  of  instruction  has  already  taken 
the  common  core  course. 

An  important  consideration  of  this  plan  is  the  requirement  to  deliver  each 
updated  BSTS  course  as  it  is  completed.  We  also  recommend  delivering  the 
new  course  in  the  same  manner  as  the  delivery  of  the  original  course. 
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Updating 


SGE  Updating: 
Production  &  Delivery 

♦  Small  Group  Exercises 

♦  Production: 

♦  Group  by  OPORD 

♦  Prioritize  updates  within  OPORD  group 

♦  Simulation-supported  last 

♦  Delivery: 

♦  Deliver  as  updated 

♦  Replace  IAW  fielding  strategy  (educate,  deliver, 
implement) 


For  the  Staff  Group  Exercises,  the  updating  plan  would  be  to  consider  the 
exercises  in  orders  groups. 

There  are  basically4-6  different  OPORDs  that  support  the  24  SGEs.  By  updating 
them  in  their  orders  groups,  we  can  economize  on  the  effort  needed  to  be  put 
into  tactical  products  and  scenario  development. 

Once  we  have  grouped  the  SGE  by  OPORDs,  we  then  prioritize  the  update  of 
the  SGEs  within  the  group.  Depending  on  the  size  of  the  OPORD  group  and  the 
amount  of  change  that  is  required,  it  is  possible  that  all  of  the  SGEs  in  a  group 
will  be  revised  concurrently. 

The  four  simulation-supported  SGEs  should  be  updated  last,  due  to  their 
complexity  and  support  requirements. 

Similar  to  the  plan  for  the  BSTS  courses,  the  SGE  should  be  delivered  to  the 
force  as  they  are  updated  and  in  the  same  manner  that  the  original  SGEs  were 
delivered. 
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Updating 

BSE  and  BBSE  Updating: 
Production  &  Delivery 


♦ BSE/BBSE: 

♦  Production: 

♦  BSE  by  mission 

♦  BBSE  in  totality 

♦  Delivery: 

♦  Deliver  BSE  as  mission  updated 

♦  Deliver  BBSE  when  updated 

♦  Replace  IAW  fielding  strategy  (educate,  deliver, 
implement) 


The  BSE  should  be  updated  and  delivered  by  mission  (there  are  three 
missions).  This  is  possible  because  of  the  design  of  the  BSE,  which  has  an 
alternative  entry  point  at  the  start  of  planning  for  each  mission  and  does  not 
require  the  staff  to  conduct  concurrent  planning. 

The  BBSE  on  the  other  hand,  must  be  updated  and  delivered  in  totality,  because 
of  the  concurrent  planning  requirements  that  are  part  of  the  exercise. 

Again,  the  delivery  should  be  done  in  the  same  manner  as  the  original  delivery 
plan  for  these  exercises. 
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Updating 

BSTS  Updating  Effort 


Calendar 

Staff 

Weeks 

Weeks 

Front  end  analysis 

6 

45 

Common  Core  Courses 

12-16 

90-120 

Bde  and  Bn  Staff  Officer  Courses 
( 1 3  pairs) 

Total  Effort,  with  efficiencies  in 

117 

260 

concurrent  work 

68-72 

395-425 
=  8PSY 

m 


This  slide  shows  our  estimated  time  it  will  take  to  update  all  of  the  BSTS 
courses. 


The  front  end  analysis  includes:  determining  what  content  changes,  what  the 
TMS  requirements  will  be,  and  what  the  user  interface  will  be.  Our  guess  is  that 
we’ll  want  to  change  all  of  these  to  some  extent. 


The  update  time  for  the  courses  assumes  that  we’ll  change  the  user  interface  for 
each  course.  The  total  BSTS  could  be  overhauled  in  less  than  1-1/2  years  by 
an  8-person  team,  if  they’re  focused  on  this  one  task. 
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Updating 

SGE  Updating  Effort 


Calendar 

Weeks 

Staff 

Weeks 

♦  Front  end  analysis 

4 

30 

♦  20  exercises  without  simulation 

20 

150 

♦  4  exercises  with  simulation 

10 

75 

♦  Total  Effort ,  with  efficiencies  in 
concurrent  work 

26 

255 

=  5  PSY 

For  SGE,  the  effort  again  starts  with  a  front  end  analysis  to  determine  the  extent 
of  the  change  required.  With  the  front-end  analysis  information,  the  estimate  is 
for  approximately  1  week  per  SGE. 

The  total  effort  in  staff  years  is  about  5  years;  the  work  could  be  accomplished  in 
6  months  be  a  team  of  10  people,  if  they  aren’t  also  doing  the  BSTS  at  the  same 
time! 
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Updating 

BSE/BBSE  Updating  Effort 


Calendar 

Weeks 

♦  BSE: 

22 

♦  Front  end  analysis  -  4  weeks 

♦  1  st  mission  -10  weeks 

♦  2nd  &  3rd  missions  -  4  weeks  each 

♦  BBSE: 

14 

♦  Front  end  analysis  -  4  weeks 

♦  3  mission  package  -10  weeks 

♦  Total  Effort 

36 

'■i v ,  vS'  » 
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This  slide  shows  the  expected  effort  for  the  BSE  and  BBSE  updates.A  team  of  6 
could  accomplish  the  changes  in  just  over  a  year,  if  they  were  not  also  doing  the 
BSTS  or  SGE. 
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The  skills  of  the  personnel  assembled  to  form  the  update  team  are  critical. 


The  requirements  shown  here  should  not  be  taken  lightly.  What  is  not  shown  is 
that  the  estimates  are  based  on  the  premise  that  the  team  doing  this  work  is 
knowledgeable  about  the  original  products. 


One  other  consideration,  is  that  the  update  work  needs  to  be  coordinated  so  we 
don’t  get  updated  products  that  disagree  with  each  other.  At  the  same  time, 
those  estimates  don’t  assume  time-sharing  between  product  efforts  can  happen 
to  any  great  degree. 
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Support  the  Force 


The  strategy  also  address  HOW  to  keep  the  products  viable. 


The  first  solution  is  to  maintain  a  link  to  the  force.  There  are  two  key  ways  to 
accomplish  that,  involving  personnel  support  at  Ft.  Knox  and  on-site  at  the  user 
units’  locations. 
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Support 


Support  the  Force 


♦  Ft.  Knox 

♦  Establish  a  HELP  DESK 

♦  1 -800-TRAINING 

♦  24  HOUR  ACCESS  to  the  Force  (Phone  &  Internet 
Access) 

♦  Menu  Driven  Options  (For  COBRAS  press  1 1 ) 

♦  Branch  School  linkage 

♦  Product  Improvement  Site  (Voice  Mail  &  Internet) 


♦  On-site  (selected  locations) 

♦  Training  Support  Coordinator 

♦  Surge  Team  support  -  as  needed 

3/ 1 


The  recommendation  for  the  Fort  Knox  component  has  several  parts.  Most  of 
the  efforts  could  be  done  efficiently  using  the  manpower  already  existing  within 
DTDD  and  its  contractors. 

The  on-site  support  recommendation  is  based  on  data  from  the  ISAT  effort, 
which  suggest  that  the  on-site  training  support  coordinator  is  very  valuable  to  the 
users.  This  person  can  deal  with  most  of  the  daily  problems  and  would  know 
where  to  look  or  who  to  ask  for  the  bigger  problems.  If  you  also  look  at  the 
SIMITAR  and  STEP  programs  for  the  ARNG,  you  will  also  see  that  these 
programs  included  a  TSC  for  each  of  the  supported  brigades. 

-  Why?  Because  the  units  have  enough  problems  trying  to  deal  with  their  daily 
actions  and  they  can’t  afford  to  dedicate  a  soldier  to  keep  up  with  these 
structured  programs. 

There  will  be  times  when  a  training  support  coordinator  needs  assistance  as 
well.  One  solution  is  a  surge  effort  where  the  development/update  teams  are 
called  to  handle  a  problem,  as  needed. 
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Surge  Effort  Considerations 


FIELD 


SUSTAIN 


lli-r...: 

UNIT-CENTERED  SUPPORT 


CONTINUING  UPDATES 


Size  based  on  needs 
Personnel  qualifications 
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The  level  of  the  surge  requirement,  would  naturally,  be  dependent  on  the 
situation  they  were  asked  to  support. 

What’s  important  here  is  that  to  have  the  right  people  available  when  there  is  a 
need  to  surge. 
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Surge 

Surge  Effort  Considerations 


♦  Size  based  on  needs 

♦  FULL  SURGE  (5-10  people) 

♦  PARTIAL  SURGE  (1-4  people) 

♦  Personnel  qualifications 

♦  Team  Leader  -  Bde  Command  Experience 

♦  Military  SMEs  -  Bde  Staff  &  Bn  Command  Experience 

♦  Computer/Simulations  Expertise  -  CBI,  Janus,  BBS 


The  recommendation  includes  two  types  of  surges. 

-  Full  surges,  to  support  a  BSE  or  BBSE 
and 

-Partial  surges,  to  support  targeted  briefings  and  train-the-trainer  sessions. 

The  qualifications  for  surge  team  members  are  the  same  as  for  the  product 
update  teams.  Using  the  same  people  would  insure  that  the  feedback  and 
observations  from  actual  implementations  were  incorporated  into  updates,  and 
that  the  most  current  updating  information  was  incorporated  in  implementations. 

However,  after  the  initial  update  work  is  done,  the  sustainment  effort  won’t 
require  a  full-time  staff  of  10  people. 
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Surge 

Surge:  Where  do  the 
people  come  from? 


This  slide  shows  at  least  five  ways  the  people  that  developed  the  original  and 
updated  work  might  be  brought  in  from  projects  they  may  be  working  on  in  the 
future  to  assist  with  a  surge  requirements. 
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Sustain 

Sustain  the  Products 


♦  Staff 


I*  One  FXXI-TP  Coordinator  (Fort  Knox  based) 
♦  Surge,  as  needed 
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The  last  part  of  the  strategy  is  the  SUSTAIN  piece. 


The  approach  is  fairly  straightforward:  Within  DTDD,  there  should  be  one  person 
responsible  for  coordinating  the  effort.  And  again,  the  surge  teams  are  drawn 
from  the  Fort  Knox  based  developers  as  needed  to  do  the  sustainment  work. 
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Summary 


IMMEDIATE  ONGOING 


Force  XXI  Training  Program  products  need  to  be  — 
«►  FIELDED 
*■  UPDATED 
«►  SUSTAINED 


I  This  strategy  presents  a  long-term  integrated  and 
multifaceted  approach  to  meeting  the  needs. 
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The  requirements  described  above  demand  an  integrated,  multifaceted  plan  that 
addresses  initial  fielding  support,  immediate  updates,  continuing  maintenance, 
and  ongoing  sustainment  for  users.  The  plan  must  take  into  consideration 
primary  users  in  the  AC  and  RC  units,  as  well  as  institutional  users  and 
supporters.  Finally,  the  plan  must  ensure  that  the  personnel  who  will  carry  out 
the  activities  are  available,  accessible,  and  of  the  appropriate  levels  of  expertise 
and  experience  with  respect  to  both  doctrine  and  the  training  products 
themselves. 
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Appendix  E 

An  Approach  to  the  Fielding,  Maintenance,  and 
Sustainment  of  Force  XXI  Training  Program  Products 

Introduction 

Over  the  past  five  years,  the  Force  XXI  Training  Program  (FXXITP)  has  been  responsible 
for  the  development  of  a  variety  of  training  products  for  combined  arms  forces.  The  training 
products  have  been  used  in  a  limited  number  of  units,  but  are  not  yet  widely  fielded. 
Furthermore,  no  mechanism  is  in  place  to  ensure  that  the  products  are  upgraded  in  response  to 
changes  in  doctrine,  organizations,  and  equipment. 

The  Directorate  of  Training  and  Doctrine  Developments  (DTDD)  at  Fort  Knox  has 
articulated  a  need  for  an  integrated,  multifaceted  plan  for  fielding,  maintaining,  and  sustaining 
selected  FXXITP  products.  The  consortium  of  Human  Resources  Research  Organization 
(HumRRO),  Raytheon,  Litton  PRC,  and  TRW-S&ITG  has  prepared  this  plan  as  an  outcome  of 
the  project  entitled  Implementation  and  Support  for  Assessment  of  Force  XXI  Training  Program 
(ISAT).  On  this  project,  conducted  in  1998-1999  with  the  U.S.  Army  Research  Institute  for  the 
Behavioral  and  Social  Sciences  (ARI),  researchers  assisted  with  implementation  of  selected 
products,  and  collected  data  and  information  pertaining  to  fielding  support,  updating 
requirements,  and  training  effectiveness. 


Background 

The  FXXITP  products  represent  an  integrated  program  of  training  that  provides  guidance 
and  practice  opportunities  for  brigades  and  battalions  at  levels  from  individuals  to  small  groups 
to  staff  (single  echelon  and  multiechelon),  as  shown  in  Figure  E-l. 

The  specific  products  shown  in  the  figure  include: 

•  The  Battle  Staff  Training  System  (BSTS)  —  a  series  of  modules  for  members  of  the 
brigade  and  battalion  staffs  that  provides  individual  instruction,  practice,  and 
comprehensive  testing  on  staff  tasks  and  processes.  The  BSTS  comprises  both 
computer-based  and  paper-based  materials,  and  also  contains  a  training  management 
system  to  track  usage.  Completed  in  1996,  the  BSTS  was  partially  updated  in  1998  to 
convert  one  set  of  materials  to  completely  computer-based  mode. 

•  Staff  Group  Exercises  (SGE)  —  Twenty-four  self-contained  vignette  exercises  for  the 
brigade  staff.  Each  exercise  focuses  on  selected  members  of  the  staff  reacting  to  a  well- 
defined  problem  in  a  scenario  setting.  Problems  are  drawn  from  such  sources  as 
National  Training  Center  (NTC)  and  other  Combat  Training  Center  (CTC)  trends 
analyses  and  first-hand  experience  of  NTC  Observer/Controllers  (O/Cs).  Twenty  of  the 
exercises  require  no  simulation  support;  the  other  four  use  Janus  or  the  Brigade  and 
Below  Battle  Simulation  (BBS).  The  SGE  were  completed  in  1996,  and  in  1998  were 
modified  slightly  to  increase  usability. 

•  The  Brigade  Staff  Exercise  (BSE)  —  A  BBS-driven  structured  exercise  for  the 
commander  and  staff  of  a  combined  arms  brigade.  Three  missions  are  included.  The 


E-l 


BSE  focuses  on  staff  coordination  and  use  of  all  assets  throughout  mission  planning, 
preparation,  execution,  and  consolidation  and  reorganization.  The  BSE,  which  was 
completed  in  1996,  provides  basic  “crawl”  level  staff  practice  opportunities. 

•  The  Brigade  and  Battalion  Staff  Exercise  (BBSE)  —  A  BBS-driven  structured  exercise 
for  the  commander,  staff,  and  one  to  three  battalions  of  a  combined  arms  brigade.  The 
BBSE  incorporates  many  of  the  features  of  a  CTC  rotation,  including  24-hour 
operations;  deployed  command  posts  (CPs),  concurrent  planning  and  execution,  and  a 
robust  OPFOR.  It  focuses  on  bringing  the  brigade  to  a  level  where  they  can  derive 
maximum  benefit  from  a  CTC  rotation.  The  BBSE  was  completed  in  1998. 


Figure  E-l.  A  stepping  stone  approach  for  supporting  brigade  training  with  Force  XXI  Training 
Program  products. 

The  products  complement  live  training,  such  as  field  training  exercises  (FTXs)  and  rotations 
to  the  CTCs.  They  are  not  designed  as  single-implementation  exercises,  but  as  tools  that  can  be 
used  repeatedly  to  enhance  skill  retention  over  time  between  live  training  exercises,  to  counter 
the  effects  of  personnel  turbulence,  and  to  reinforce  lessons  learned  in  live  training  exercises. 

The  training  products  were  developed  by  members  of  the  consortium,  under  contract  to  ARI. 
Each  of  the  training  products  is  fully  supported  by  materials  for  the  training  audience,  trainers, 
training  managers,  and  simulations  managers  (as  appropriate).  However,  the  completeness  of  the 
training  packages  makes  implementation  a  significant  undertaking.  Recent  investigation  of  the 
requirements  for  implementation  suggests  that,  while  the  products  are  exportable,  assistance 
from  experienced  developers  or  trainers  is  critical  for  deriving  full  benefits  from  use  of  the 
products. 
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To  date,  the  various  products  have  been  used  by  brigades  and  battalions  at  Fort  Hood,  Fort 
Riley,  and  Fort  Lewis,  and  by  a  National  Guard  unit  in  Kentucky;  been  reviewed  by  the  16th 
Cavalry  Squadron  at  Fort  Knox;  and  are  being  considered  for  use  by  7ATC  U.S.  Army  Europe 
(USAREUR).  Users  and  reviewers  have  been  overwhelmingly  positive  about  the  utility  of  the 
products,  and  members  of  those  units  who  have  rotated  to  other  jobs  are  continually  in  contact 
with  the  FXXITP  to  request  materials  and  implementation  assistance. 

The  ISAT  project  plans  included  providing  all  of  the  product  training  support  packages 
(TSPs),  a  suite  of  BSTS  computers,  and  an  on-site  coordinator  to  support  implementation. 
Research  into  support  requirements  and  training  effectiveness  involved  intensive  observation, 
documentation,  and  direct  data  collection  from  training  participants  within  two  user  brigades. 

The  ISAT  work  is  completed  and  preliminary  analyses  of  user  reactions  reveal  three 
common  themes: 

•  The  concept  is  good,  and  the  products  need  to  be  made  available  to  units. 

•  Some  of  the  content  is  dated — initial  updates  and  continuing  maintenance  of  currency 
are  essential. 

•  Because  of  the  product  complexity,  units  and  installations  require  support  prior  to  and 
during  implementation. 

Problem 

The  problem  confronting  the  FXXITP  is  three-fold: 

•  Fielding:  The  FXXITP  products  must  be  made  available  to  combined  arms  units, 
including  both  active  component  (AC)  and  reserve  component  (RC)  units.  Widespread 
use  of  the  products  in  those  units  depends  on  support  from  and  education  within  U.S. 
Army  Training  and  Doctrine  Command  (TRADOC)  schools,  CTCs,  and  Training 
Support  Brigades  (TSBs)  and  exercise  divisions.  Information  must  also  be  provided  to 
simulations  center  personnel,  who  will  carry  the  primary  burden  for  implementation  of 
the  simulation-based  training. 

•  Maintenance:  The  training  products  already  require  analysis  and  upgrading  in  response 
to  recent  changes  in  doctrine,  organizations,  and  equipment.  This  need  is  immediate,  but 
such  updating  will  also  be  required  on  a  continuing  basis,  to  ensure  that  the  products 
remain  current.  The  maintenance  should  address  not  only  doctrine,  organizations,  and 
equipment,  but  also  instmctional  and  simulation  technologies. 

•  Sustainment:  For  the  first  year  of  use,  units  will  require  support  during  implementation. 
Furthermore,  procedures  and  support  for  adapting  the  SGE,  BSE,  and  BBSE  to  specific 
unit  training  needs  will  help  to  insure  widespread  use  and  maximum  training  benefit. 
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Figure  E-2  illustrates  our  understanding  of  the  relationships  among  the  processes.  As 
shown,  initial  fielding  gradually  transitions  to  ongoing  sustainment.  Similarly,  initial  updates  are 
succeeded  by  a  process  of  continuing  updates.  Feedback  and  lessons  learned  during  fielding  are 
used  to  make  the  initial  updates,  which  are  then  delivered  and  incorporated  into  the  already 
fielded  products.  This  process  continues  over  time:  Units  use  the  products  and  provide 
suggestions  or  concerns,  and  developers  use  the  information  to  make  continual  improvements. 
Although  solutions  for  each  part  of  the  process  can  be  planned  and  executed  separately,  all  three 
needs  must  be  addressed  in  order  for  any  solution  to  be  effective. 


Figure  E-2.  Integration  of  fielding,  maintenance,  and  sustainment  activities  for  Force  XXI 
Training  Program  products. 


Plan  for  Fielding,  Maintenance,  and  Sustainment 

This  section  of  the  plan  includes  our  approach  to  all  three  facets  of  the  problem:  fielding, 
maintenance,  and  sustainment.  It  is  an  integrated,  multifaceted  plan  that  relies  on  experienced 
training  developers  to  support  the  fielding  and  sustainment  on  an  as-needed  basis,  while  also 
accomplishing  the  updates  to  and  maintenance  of  the  products.  Each  of  the  three  facets  is 
addressed  separately  below,  followed  by  the  integrated  plan  for  accomplishing  all  three 
requirements  with  a  team  organized  for  efficiency  and  effectiveness. 

Support  for  Fielding  Requirements 

As  described  in  the  IS  AT  report,  fielding  requires  support  on  two  levels: 

•  Information  for  product  users,  managers,  and  personnel  in  related  training  arenas,  and 

•  Assistance  for  users  and  managers  as  part  of  first-use  implementation. 
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Information  Activities 


In  order  for  the  products  to  be  used,  some  amount  of  information  is  needed.  The  plan  for 
educating  the  force  has  to  include  the  audience  for  the  products,  beginning  with  the  senior 
leaders  in  the  units.  But  there  are  others  who  must  also  be  the  subject  of  targeted  education 
efforts,  including: 

•  Senior  leaders  of  the  units’  higher  headquarters,  who  will  need  to  support  use  of  the 
products.  Their  support  includes  not  only  encouragement,  but  also  personnel  support 
during  implementation. 

•  Senior  leaders  at  TRADOC  schools,  where  the  products  may  be  implemented  in 
advanced  officer  and  NCO  courses. 

•  Managers  of  the  simulation  centers,  since  they  are  the  key  implementers  of  two  of  the 
products. 

•  Training  Support  Brigades  (TSBs),  Regional  Training  Brigades  (RTBs),  and  exercise 
divisions  who  support  implementation  or  recommend  use  of  the  products. 

•  Units  and  agencies  in  the  force  that  may  never  use  the  product,  but  need  to  know  about 
the  product  in  order  to  make  recommendations  to  user  units.  These  include  the  CTCs, 
the  Battle  Command  Training  Program  (BCTP),  and  the  pre-command  course  (PCC). 

Each  of  these  audiences  requires  a  different  level  of  information.  For  some,  a  general 
information  overview  that  describes  the  products  and  their  specific  niche  in  the  overall  training 
strategy,  will  be  appropriate.  These  overviews  must  be  repeated  with  some  frequency  (e.g., 
annually)  as  personnel  turnover  occurs. 

For  others,  more  detailed  information  must  be  provided.  Information  sessions  that  deliver 
talk-through/walk-through  tours  of  specific  components  of  the  products  must  be  conducted  as 
follow-up  to  the  general  overview.  These  sessions  will  ensure  that  individuals  with  no  prior 
experience  with  the  products  will  learn  enough  to  make  informed  decisions  about  product  use. 

The  last  part  of  the  education  process  involves  the  on-site  education  that  must  occur  at  the 
using  unit  site  once  they  receive  the  materials.  This  education  must  include: 

•  how  the  products  are  related  and  how  use  of  all  products  will  support  the  brigade’s 
training  strategy, 

•  how  each  product  and  its  TSP  components  are  intended  to  be  used, 

•  what  the  implementation  schedule  includes, 

•  what  preparation  is  required,  and 

•  how  each  product  can  be  modified  and  some  of  the  problems  that  can  result  from 
incomplete  modifications. 
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Approach:  Information  Activities 


The  approach  to  the  requirement  for  providing  information,  as  propounded  in  this  plan, 
involves  a  series  of  four  types  of  activities: 

•  Information  briefings 

•  Information  seminars 

•  Targeted  briefings 

•  On-site  seminars. 

Information  briefings.  The  first  activity,  providing  a  general  information  overview,  takes 
the  form  of  an  information  briefing  at  the  Armor  Conference,  Infantry  Conference,  and  other 
conferences  such  as  the  Association  of  the  U.S.  Army  (AUSA).  The  briefings  themselves  should 
be  presented  by  the  Director  of  DTDD.  Because  the  information  briefings  will  be  open  to  all 
attendees  at  these  conferences,  a  large  audience  would  be  expected.  However,  the  targeted  group 
for  the  briefing  will  be  the  senior  leaders  of  installations  and  units  where  the  products  would  be 
used,  both  AC  and  RC.  Specific  invitations  should  be  sent  to  these  individuals,  over  the 
signature  of  the  DTDD  Director. 

Information  seminars.  An  essential  follow-on  to  the  information  briefing  would  be  focused 
information  seminars  to  provide  more  detailed  information  about  each  of  the  products.  These 
seminars  should  be  presented  immediately  following  the  information  briefing,  at  the  conference 
sites.  Six  such  seminars  are  envisioned,  covering: 

•  BSTS  components  and  implementation  requirements 

•  SGE  components  and  implementation  requirements 

•  BSE/BBSE  components 

-  Exercise  Preparation 

-  Exercise  Control 

-  Performance  Objectives 

-  Simulation  Support 

As  with  the  information  briefings,  the  seminars  would  be  open  to  all  conference  attendees. 
The  target  of  the  seminars  would  be  the  individuals  who  will  ultimately  be  responsible  for 
implementation.  Because  the  seminars  are  focused,  individuals  could  select  one  or  more  that 
address  their  own  concerns.  For  example,  the  RTB  personnel  will  likely  be  most  interested  in 
the  seminar  on  performance  objectives,  while  simulation  center  managers  would  be  most 
interested  in  simulation  support. 

Targeted  briefings.  These  briefings  include  both  a  generic  briefing  and  tailored  versions  of 
the  information  briefing  described  above.  The  generic  briefing  should  be  updated  periodically, 
and  may  be  used  by  DTDD  whenever  there  are  targets  of  opportunity  for  talking  about  the 
FXXITP  products. 
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The  tailored  versions  would  be  used  for  groups  such  as  CTC  personnel,  the  Combined  Arms 
Center  (CAC),  leaders  of  TRADOC  schools,  PCC  groups,  and  staff  training  courses  (such  as 
Command  and  General  Staff  Officer  Course,  the  Combined  Arms  and  Services  Staff  School,  and 
Armor  and  Infantry  advanced  courses  for  officers  and  NCOs). 

Most  of  these  briefings  should  be  delivered  by  the  Director  of  DTDD  or  his  designated 
representative. 

On-site  seminars.  The  same  seminars  that  were  provided  with  the  information  briefings 
should  also  be  provided,  in  a  reduced  form,  at  implementation  sites.  On-site  seminars  will  be 
needed  in  cases  where  installation  and  unit  personnel  were  unable  to  attend  conference  briefings 
and  seminars.  Because  these  seminars  would  be  conducted  on-site,  the  specific  implementation 
constraints  of  the  particular  unit  or  simulation  center  could  be  addressed  in  these  sessions. 

Each  on-site  seminar  should  take  the  form  of  a  train-the-trainer  session.  The  target  audience 
for  the  BSTS  would  be  unit  trainers  and  those  charged  with  managing  the  system.  For  the  SGE, 
the  brigade  executive  officer  (XO)  and  training  officer  (S3)  should  participate,  and  additional 
training  should  be  provided  to  the  simulation  center  manager  for  use  of  the  four  simulation- 
supported  exercises. 

Both  the  BSE  and  the  BBSE  should  be  covered  in  a  single  series  of  four  seminars.  These 
seminars  would  be  addressed  to  the  brigade  and  division  training  managers  (S3  and  G3)  or 
installation  scheduling  and  resourcing  agency  for  discussions  of  exercise  resources  and 
preparation,  exercise  control,  and  training  and  preparation  for  the  observers  and  training 
audience.  A  separate  session  could  address  simulations  and  workstation  training. 

First-Use  Assistance 


Once  a  decision  has  been  made  to  implement  one  of  the  FXXITP  products,  more  intensive 
support  should  be  provided.  During  the  weeks  leading  up  to  implementation,  a  wide  variety  of 
decisions  and  activities  is  required.  Early  decisions  are  relatively  simple,  but  decisions  and 
activities  that  occur  immediately  prior  to  implementation  are  both  critical  and  bewildering  for  a 
first-time  user.  Experience  in  implementing  the  products  during  the  ISAT  project  indicates  that 
the  using  units  need  an  increasing  level  of  assistance  over  the  weeks  leading  up  to 
implementation  in  order  to  fully  understand  and  utilize  the  products. 

Approach:  First  Use  Assistance 

Assistance  for  first  use  of  the  products  will  occur  over  a  long  period  of  time,  and  the  need 
for  assistance  will  increase  as  implementation  approaches.  In  order  to  address  the  early 
questions  and  concerns  efficiently,  training  support  coordinators  should  be  resident  at  the 
installations  where  the  products  are  to  be  used.  The  training  support  coordinators,  who  should  be 
experienced  developers  and  implementers  of  the  products,  will  be  the  first  source  for  the  units 
when  they  have  questions  or  concerns.  These  coordinators  would  be  backed  up  by  other 
developers  at  Fort  Knox  who  would  be  primarily  engaged  in  the  maintenance  activities  described 
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below.  Thus,  any  question  or  issue  that  the  coordinator  was  not  able  to  answer  could  be  referred 
to  the  larger  group  of  experts. 

This  expertise  would  be  available  to  the  coordinators  by  simple  means  of  telephone,  e-mail, 
video-teleconferencing,  and  via  the  FXXITP  Internet  site,  which  should  have  a  training 
assistance  page.  The  same  expertise  and  assistance  would  also  be  available  for  users  who  do  not 
have  an  on-site  coordinator,  such  as  RC  units  and  TRADOC  schools. 

For  the  BSTS  and  SGEs,  the  on-site  support  and  Fort  Knox-based  expertise  would  be 
sufficient  to  support  use.  For  the  BSE  and  BBSE,  a  surge  team  capability  is  proposed.  The 
surge  team  would  be  formed  as  needed,  comprising  2-6  experts  from  Fort  Knox  and  spending 
anywhere  from  a  few  days  to  a  few  weeks  with  units  prior  to  and  during  implementation.  Their 
role  will  include  advising  the  simulation  center  manager  on  matters  relating  to  the  scenario  files 
and  workstation  training;  assisting  with  pre-exercise  decisions  by  the  commander,  unit 
preparation,  and  observer  training;  and  guiding  the  exercise  director,  division  response  cell,  and 
OPFOR  controller  in  conducting  the  exercise. 

Support  for  Maintenance  Requirements 

As  evidenced  by  comments  from  users  and  reviewers,  the  FXXITP  products  are  in  need  of 
updating.  While  there  is  an  immediate  need,  updating  should  be  done  on  a  continuing  basis. 
Additionally,  such  updating  must  be  done  concurrently  with  the  fielding  and  implementation 
discussed  above. 

The  process  for  updating  involves  two  tasks: 

•  Develop  a  plan  for  making  and  fielding  the  immediate  updates 

•  Develop  a  plan  for  effecting  continual  maintenance. 

The  immediate  updating  plan,  the  ongoing  maintenance  plan,  and  the  fielding  plan  described 
above  should  be  integrated  in  order  to  provide  products  that  the  units  and  schools  can  use. 

Immediate  Updating  and  Fielding 

For  all  of  the  FXXITP  products,  the  first  emphasis  for  updating  should  be  to  align  the 
products  with  current  doctrine,  organizations,  and  equipment.  Additionally,  advances  in 
instructional  technologies  and  simulation  capabilities  demand  concomitant  changes  in  the  way 
the  training  is  conducted.  One  very  immediate  example  concerns  the  year  2000  (Y2K)  problem. 
For  all  of  the  technology-driven  training  products,  the  technologies  themselves  must  be 
examined  for  Y2K  readiness,  and  the  products  must  be  modified  if  necessary  to  ensure  readiness. 
Other  examples  include  advances  in  Internet  capabilities,  upgrades  to  simulations  such  as  BBS 
and  Janus,  introduction  of  new  simulations  such  as  Close  Combat  Tactical  Trainer  (CCTT),  and 
even  the  propensities  at  particular  locations  to  use  only  BBS  or  Janus.  It  will  be  crucial  to 
identify  the  updates  that  are  required  immediately,  as  opposed  to  those  that  can  wait  for  a  second 
or  third  wave  of  change. 
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The  implementation  described  earlier  should  not  wait  for  all  of  the  initial  updates  to  be 
made.  Unit  trainers  are  able  to  identify  deviations  from  current  doctrine  and  make  the  necessary 
adjustments  on  a  short-term  basis  with  the  sustainment  support  described  above.  However,  the 
immediate  updates  should  be  addressed  as  soon  as  possible.  As  the  updates  are  made,  they  must 
be  incorporated  into  the  already-fielded  products  in  a  way  that  causes  minimal  disruption. 

Approach:  Immediate  Updates  and  Fielding 

Even  as  fielding  is  ongoing,  the  update  team  should  be  actively  searching  out  feedback  from 
units  using  the  materials,  for  use  in  planning  for  and  carrying  out  the  most  urgently  needed 
updates.  Some  of  the  materials  within  the  products  are  based  on  1995  doctrine,  organizations, 
and  battlefield  systems.  Many  changes  that  are  needed  in  these  areas  are  already  documented,  as 
a  result  of  work  on  the  IS  AT  project.  The  Army’s  standard  graphics  systems  have  changed, 
rendering  parts  of  the  BSTS  dated,  though  not  useless.  The  BSE  division  organization  includes 
four  brigades,  each  with  four  battalions — a  logical  organization  for  the  separate  enhanced 
brigade  for  which  the  training  was  originally  planned,  but  no  longer  suitable  for  any  existing 
brigade  structures. 

Other  changes,  outside  the  military,  also  require  that  the  products  be  revised.  The 
technologies  for  computer-based  instructional  systems  are  much  more  complex  and  powerful 
than  were  the  technologies  of  the  mid-90s.  The  simulations  that  drive  the  BSE  and  BBSE  have 
been  updated  since  the  exercises  were  developed,  and  the  materials  need  to  be  examined  and 
modified  to  be  usable  on  those  simulations.  Delivery  of  training  support  package  materials  has 
always  depended  heavily  on  print  mode,  although  recently  CD-ROMs  have  been  used. 

However,  as  Internet  capabilities  continue  to  expand,  the  use  of  internet-downloadable  and 
internet-interactive  training  should  be  explored. 

A  team  that  comprises  FXXITP  product  developers,  with  a  thorough  understanding  of  the 
FXXITP  products,  their  components,  changes  to  military  doctrine  and  systems,  and 
implementation  concerns  and  constraints,  should  be  formed  to  support  the  updating.  Stationed  at 
Fort  Knox,  they  would  be  responsible  for  the  updates  and  trials  of  the  updates,  while  also  being 
available  to  form  up  the  surge  teams  for  initial  implementation  support.  The  initial  assessment  of 
the  immediate  changes  required  to  the  products,  as  detailed  in  the  ISAT  report,  results  in  the 
approach  presented  here  for  each  of  the  four  products. 

Battle  Staff  Training  System.  For  the  BSTS,  updating  should  focus  first  on  the  common 
core  modules  at  both  brigade  and  battalion  levels.  These  modules  are  a  part  of  the  training  for  all 
members  of  the  staff,  and  therefore  have  the  widest  applicability.  Priority  should  go  next  to  the 
brigade  and  battalion  commanders’  course  and  the  courses  for  the  primary  staff  (XO,  SI,  S2,  S3, 
S4,  and  FSO)  at  both  levels.  The  remaining  courses  should  then  also  be  updated.  Throughout 
the  process,  the  courses  at  brigade  and  battalion  level  should  be  paired  for  examination  and 
modification. 

Initial  changes  to  the  BSTS  should  focus  on  four  areas:  Y2K  compliance,  doctrinal 
currency,  100%  computer-based,  and  fielded  via  the  Internet.  Because  current  computer  systems 
are  not  yet  advanced  enough  to  make  Internet-interactive  training  feasible,  the  first  complete  set 
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of  updated  materials  would  instead  be  Internet-downloadable.  The  updated  materials  would  be 
loaded  on  the  Internet  and  made  available  to  users  as  they  are  developed,  without  waiting  for 
completion  of  the  update  process  across  all  courses. 

At  the  same  time,  exploration  of  technologies  that  may  make  full  internet-interactive  courses 
possible  should  continue,  and  efficient  ways  to  allow  for  distribution  of  overlays  in  BSTS  should 
be  identified.  Finally,  a  training  management  strategy  must  be  developed  that  will  allow  users  to 
record  their  successful  completion  of  courses,  provide  training  managers  with  up-to-date 
information  about  staff  use  of  the  BSTS,  and  ensure  smooth  transitions  from  the  currently  fielded 
materials  to  the  updated  materials. 

The  updating  cannot  be  done  in  a  vacuum.  Two  important  sources  of  information  should  be 
the  training  developers  and  doctrine  developers  within  DTDD,  and  the  Automated  Systems 
Approach  to  Training  (ASAT).  The  databases  within  ASAT  contain  a  wealth  of  information 
about  task  analysis,  and  it  will  be  important  to  link  the  BSTS  training  objectives  to  the  task 
information  provided  in  ASAT. 

Staff  Group  Exercises.  The  immediate  need  for  the  SGE  is  to  verify  the  currency  and 
applicability  of  the  materials.  The  SGE  materials  were  originally  developed  in  two  efforts.  The 
first  12  exercises  were  designed  to  address  the  training  needs  of  1 1  selected  members  of  the 
brigade  staff,  and  like  the  BSE  were  developed  with  the  separate  enhanced  brigade  structure  in 
mind.  The  remaining  12  exercises  expanded  the  training  audience  to  16  members  of  the  staff, 
and  were  based  on  a  more  typical  division  and  brigade  organization.  Therefore,  the  first  area  for 
review  will  be  the  appropriateness  of  the  SGE  with  respect  to  the  unit  organization.  Another 
pressing  need  is  to  reexamine  the  original  12  exercises,  to  see  which  should  be  expanded  in  order 
to  provide  additional  training  to  the  expanded  training  audience. 

The  individual  exercises  should  be  updated  in  orders  groups.  Across  the  24  exercises,  there 
are  between  four  and  seven  identical  or  nearly  identical  division  orders  that  underlie  the  tactical 
situations.  The  exercises  should  be  grouped  by  OPORDs,  and  then  the  update  of  the  SGEs 
within  the  group  must  be  prioritized.  Depending  on  the  size  of  the  OPORD  group  and  the 
amount  of  change  that  is  required,  it  is  possible  that  all  of  the  exercises  in  a  group  will  be  revised 
concurrently.  The  four  simulation-supported  exercises  should  be  tackled  last,  due  to  the 
complexity  of  these  exercises  and  their  support  requirements. 

Other  aspects  of  SGE  updating  should  mirror  the  considerations  stated  above  for  the  BSTS. 
The  ASAT  will  be  used  for  doctrinal  information  and  task  referents,  and  it  will  be  necessary  to 
ensure  that  revised  SGE  materials  can  be  loaded  into  the  ASAT  TSP  library.  As  with  the  BSTS 
courses,  the  SGE  should  be  delivered  to  the  force  as  they  are  updated.  Internet-downloadable 
delivery  will  be  used,  and  an  efficient  means  for  providing  the  overlays  must  also  be  identified. 

Brigade  Staff  Exercise.  The  BSE  is  due  for  a  serious  overhaul.  Already  it  is  known  that  the 
division  and  brigade  organizations  are  atypical,  the  language  and  guidance  surrounding  the 
military  decision-making  process  are  out-of-date,  and  widely  used  equipment,  such  as  the  All 
Source  Analysis  System  (ASAS),  are  not  included  in  exercise  conditions.  Additionally,  the 
linkage  between  BSE  and  ASAT  task  databases  has  not  yet  been  definitively  delineated. 
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Update  and  delivery  of  the  BSE  should  be  accomplished  by  mission.  This  will  be  possible 
because  of  the  BSE  design  structure,  which  provides  discrete  entry  points  for  each  mission  and 
does  not  require  the  staff  to  conduct  concurrent  planning.  As  a  result,  the  three  missions  are 
separable,  and  fielding  of  updated  materials  can  be  done  as  each  mission  is  completed. 

Brigade  and  Battalion  Staff  Exercise.  Because  it  is  the  most  recently  completed  of  the 
FXXITP  products,  the  BBSE  is  also  the  most  current.  Nonetheless,  the  underlying  assumptions 
should  be  examined  in  light  of  current  doctrine,  organization,  and  equipment.  The  BBSE  should 
be  updated  as  a  single  unit,  because  the  three  mission  segments  are  not  separable. 

On-Going  Maintenance 

Assuring  currency  is,  of  course,  a  losing  proposition.  The  training  products  will  always  lag 
doctrinal  and  technology  advances.  The  goal  must  be  to  minimize  the  lag,  while  ensuring 
smooth  introduction  of  the  changes  into  training  products  that  are  in  widespread  use.  This  will 
require  judgements  concerning  the  urgency  of  the  changes,  decisions  on  how  often  and  by  what 
means  the  changes  should  be  communicated  to  the  users,  and  evaluation  of  the  efficacy  of  the 
procedures. 

Approach:  On-Going  Maintenance 

As  the  fielding  and  implementation  support  described  earlier  are  ongoing,  the  Fort  Knox- 
based  update  team  should  be  amassing  observation,  feedback,  and  lessons  learned  concerning  the 
products  and  their  use.  This  information  will  be  needed  to  determine  the  needs  for  additional 
maintenance  to  the  products.  There  are  a  number  of  needs  that  have  already  been  identified  by 
the  ISAT  team,  as  described  below. 

One  such  need  for  BSTS  and  SGE  is  to  implement  the  training  in  a  distributed  Internet- 
interactive  mode.  The  technology  will  soon  be  adequate  to  support  such  training,  although  the 
supporting  hardware  and  software  may  not  be  widely  available  to  users  yet.  When  the  time  is 
right,  the  modified  delivery  system  should  allow  the  FXXITP  to  move  away  from  printed 
instructions  for  SGE  to  more  user-friendly  methods.  The  reading  burden  within  SGE  is  not 
excessive:  The  training  coordinator  can  read  and  understand  the  administrative  instructions  and 
guidance  in  about  an  hour,  on  paper  or  on  a  computer  display  screen.  However,  the  task  for  the 
training  audience  of  comprehending  the  tactical  situation  and  conditions  can  be  complex.  Other 
ways  of  conveying  the  information  to  the  training  coordinator  and  the  training  audience  should 
be  considered. 

A  related  need  concerns  the  amount  of  printed  material  required  for  BSE  and  BBSE 
implementation.  Currently,  all  of  the  TSPs  for  these  exercises  are  provided  as  printed  material. 
Research  should  continue  on  ways  to  reduce  the  reading  burden  while  still  providing  all  of  the 
necessary  information. 

During  the  course  of  the  past  five  years,  there  have  been  developments  in  instructional 
theory  and  practice  that  do  not  involve  technologies.  Just  within  the  FXXITP  work,  developers 
have  learned  about  the  learning  styles  of  military  audiences,  the  optimal  organization  of  training 
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support  materials,  and  the  types  of  guidance,  supporting  information,  and  job  aids  that  should  be 
included.  The  BBSE  incorporates  some  of  the  innovative  ways  of  conducting  training,  in  the 
form  of  performance  objectives  that  include  techniques  and  procedures.  The  BSE  should  be 
further  updated  to  implement  BBSE  lessons  learned  in  the  materials. 

Other  maintenance  needs  for  the  products  are  the  result  of  changes  within  the  Army.  The 
introduction  of  digital  systems  and  organizations  requires  production  of  TSPs  that  incorporate 
those  capabilities.  Already  DTDD  and  ARI  have  monitored  development  of  a  plan  for 
converting  the  products.  Other  terrain  areas  should  also  be  incorporated  into  modified  exercises. 
Simulations  are  frequently  upgraded,  and  the  products  should  be  modified  in  order  to  make  them 
still  usable.  Products  that  are  currently  implemented  within  one  simulation  should  be  converted 
to  versions  using  simulations  available  at  other  locations. 

Support  for  Sustainment  Requirements 

Experience  on  the  ISAT  project  indicates  that,  once  implementation  has  occurred  for  a  unit, 
the  assistance  needed  for  subsequent  implementations  is  markedly  reduced.  The  level  of 
assistance  needed  will  vary  across  the  training  products.  An  integrated  plan  is  needed  that  will 
ensure  that  the  needed  support  is  available  for  units  and  institutions  as  they  continue  to  use  the 
products.  This  focus  on  unit-centered  training  is  at  the  core  of  the  sustainment  need. 

Unit-Centered  Training  Support 

The  need  for  sustainment  will  vary  for  the  products.  The  BSTS  is  used  continually,  so  that 
the  experience  gained  concerning  implementing  and  managing  the  training  is  constantly 
reinforced.  Similarly,  in  the  case  of  the  SGE,  continual  unit  use  will  insure  continuity  of 
implementation  expertise.  External  support  may  be  required  when  upgrades  are  made,  and  in  the 
event  of  a  software  glitch  or  damaged  product  materials. 

The  BSE  and  BBSE  are  more  likely  to  require  sporadic  intensive  support.  In  most  cases,  the 
TSPs  will  reside  with  the  simulation  support  center,  and  those  personnel  will  be  responsible  for 
managing  and  assisting  with  implementation.  However,  it  is  expected  that  the  BSE  and  BBSE 
will  be  used  less  frequently  than  the  BSTS  and  SGE.  Therefore,  it  will  be  necessary  to  plan  to 
provide  support  for  one  or  more  subsequent  implementations. 

There  is  another  consideration  for  on-going  support  of  the  BSE  and  BBSE,  concerning 
locally-directed  modifications.  Over  the  past  three  years,  every  implementation  of  the  exercises 
has  involved  some  degree  of  customization.  In  each  case,  exercise  developers  assisted  in  making 
the  modifications  throughout  the  TSPs  in  order  to  ensure  the  structural  integrity  of  the  product. 

It  is  anticipated  that  units  will  continue  to  want  to  impose  changes  on  the  exercises  prior  to 
implementation.  Therefore,  it  will  be  very  important  to  have  a  procedure  for  providing 
continuing  advice  and  support  for  units  and  simulation  support  center  personnel,  probably  for  the 
first  three  or  four  implementations  at  a  given  location. 
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